Search
Recent Tweets

Entries in .net (29)

Friday
May242013

ASP.NET NLog Sql Server Logging and Error Handling Part 2

Be sure to check out Part 1 as this post builds upon it and the two go hand in hand.

Series Overview

Part 1 - Setting up logging with ASP.NET MVC, NLog and SQL Server

Part 2 - Unhandled exception processing, building an error report, emailing errors, and custom error pages.

Custom Error Handling Attribute

Added in FilterConfig.RegisterGlobalFilters and bound in DiagnosticModule, AppErrorHandlerAttribute invokes reporting the unhandled exception and setting the error view to be displayed to the end user. An enableErrorPages appSetting controls whether any of this is done; for local debugging or a dev web server having this off might be desirable.
namespace NLogSql.Web.Infrastructure.ErrorHandling
{
    public class AppErrorHandlerAttribute : FilterAttribute, IExceptionFilter
    {
        [Inject]
        public IErrorReporter Reporter { get; set; }

        public void OnException(ExceptionContext exceptionContext)
        {
            if (exceptionContext.ExceptionHandled) return;

            if (ConfigurationManager.AppSettings["enableErrorPages"] == "false")
            {
                AppLogFactory.Create<AppErrorHandlerAttribute>().Error(
                    "Unexpected error. enableErrorPages is false, skipping detailed "
					+ "error gathering. Error was: {0}",
                    exceptionContext.Exception.ToString());
                return;
            }

            Ensure.That(Reporter, "Reporter").IsNotNull();
            Reporter.ReportException(exceptionContext);

            SetErrorViewResult(exceptionContext);
        }

        private static void SetErrorViewResult(ExceptionContext exceptionContext)
        {
            var statusCode = new HttpException(null, exceptionContext.Exception)
				.GetHttpCode();

            exceptionContext.Result = new ViewResult
            {
                ViewName = MVC.Shared.Views.ViewNames.Error,
                TempData = exceptionContext.Controller.TempData,
                //ViewData = new ViewDataDictionary<ErrorModel>(new ErrorModel())
            };

            exceptionContext.ExceptionHandled = true;
            exceptionContext.HttpContext.Response.Clear();
            exceptionContext.HttpContext.Response.StatusCode = statusCode;
            exceptionContext.HttpContext.Response.TrySkipIisCustomErrors = true;
        }
    }
}

Logging and Reporting the Error

In the same namespace the ErrorReporter class invokes generation of an error report and logs and emails the error report. The overload with customActivityMessage would generally be used with handled exceptions where it may still be desirable to report the exception in cases.
public class ErrorReporter : IErrorReporter
{
	private readonly ILog _log;

	public ErrorReporter(ILog log)
	{
		_log = Ensure.That(log, "log").IsNotNull().Value;
	}

	private string CustomActivityMessage { get; set; }

	public void ReportException(ControllerContext controllerContext, 
		Exception exception, string customActivityMessage = null)
	{
		this.CustomActivityMessage = customActivityMessage;
		ReportException(new ExceptionContext(controllerContext, exception));
	}

	public void ReportException(ExceptionContext exceptionContext)
	{
		var errorInfo = new ErrorReportInfo(exceptionContext, this.CustomActivityMessage);
		errorInfo.Generate();
		_log.Error("Unexpected error: {0}", errorInfo.ReportText);

		if (errorInfo.Errors.Any())
			_log.Error("Error generating error report. Original exception: {0}", 
				exceptionContext.Exception);

		// sending mail can be a little slow, don't delay end user seeing error page
		Task.Factory.StartNew(
		state =>
		{
			var errorReport = (ErrorReportInfo)state;
			DependencyResolver.Current.GetService<IErrorEmailer>()
				.SendErrorEmail(errorReport);
		},
		errorInfo).ContinueWith(t =>
		{
			if (null != t.Exception)
				_log.Error("Error sending email: " + t.Exception.ToString());    
		});
	}
}

Building the Error Report

Various diagnostic info classes are responsible for building different diagnostic "sub reports". Each inherits from DiagnosticInfoBase which is a glorified StringBuilder, with functionality to build both a plain text version of the report (logged to DB) as well as an HTML formatted version (used for emails).

The base class has safe appending and formatting functionality and ensures that any error in generating a part of the report doesn't stop the whole process.
A sample implementation:
public class FormInfo : DiagnosticInfoBase
{
	private readonly HttpRequestBase _request;

	public FormInfo(HttpRequestBase request)
	{
		_request = request;
	}

	protected override void GenerateReport()
	{
		StartTable();
		var keys = _request.Form.AllKeys.OrderBy(s => s).ToList();

		foreach (var name in keys)
		{
			var value = _request.Form[name];

			if (null != value && name.Contains("password", StringComparison.OrdinalIgnoreCase))
			{
				value = new string('*', value.Length);
			}

			AppendRow(name, value);
		}

		EndTable();
	}
}
The ErrorReportInfo class combines the output of each section for the full error report.



In the end this produces a sample HTML email report like this.

Sending the Email

Sending the email logs the result on failure or success,
public interface IErrorEmailer
{
	void SendErrorEmail(ErrorReportInfo errorInfo);
}

public class ErrorEmailer : IErrorEmailer
{
	private readonly IMailer _mailer;
	private readonly ILog _log;

	public ErrorEmailer(IMailer mailer, ILog log)
	{
		_mailer = Ensure.That(mailer, "mailer").IsNotNull().Value;
		_log = Ensure.That(log, "log").IsNotNull().Value;
	}

	public void SendErrorEmail(ErrorReportInfo errorInfo)
	{
		try
		{
			var subject = string.Format("{0} Error", 
				Assembly.GetExecutingAssembly().GetName().Name);

			if (null != errorInfo.Server && null != errorInfo.Location
				&& !string.IsNullOrWhiteSpace(errorInfo.Location.ControllerAction)
				&& !string.IsNullOrWhiteSpace(errorInfo.Server.HostName))
			{
				subject = string.Format("{0}: {1} - {2}", subject, 
					errorInfo.Server.HostName, errorInfo.Location.ControllerAction);
			}

			var to = AppSettings.Default.Email.ErrorMessageTo;
			_mailer.SendMail(to, subject, errorInfo.ReportHtml);

			_log.Info("Sent email: {0} to {1}", subject, to);
		}
		catch (Exception ex)
		{
			_log.Error("Error sending error report email: {0}", ex);
		}
	}
}

The Error View

At the end of AppErrorHandlerAttribute the SetErrorViewResult invoked the Views\Shared\Error.cshtml page. According to the HTTP status code, content such as text, images and styles differ. Separate error pages may have more advantages if there are a larger number of differences; a single page was less work here.
@section styles{
    <link rel="stylesheet" href="~/Content/Error.css"/>
}

@{
    var statusTitleMap = new Dictionary<int, string> {
             {404, "Something got lost in the shuffle"},
             {410, "Gone like yesterday"},
             {500, "Something go boom"}
         };
}

@section hero{
    <div id="error-body" class="container-fluid">
        <div class="row-fluid">
            <div class="span3">
                @{ var errorClass = (Response.StatusCode == 404 || Response.StatusCode == 410) ? "_" + Response.StatusCode : "_500";}
                <div id="errorImageBlock" class="@errorClass"></div>
            </div>
    
            <div class="span8 offset1">
                <div class="row-fluid">
                    <div class="span12 text-center" id="status-code">
                        @Response.StatusCode
                    </div>
                </div>
                <div class="row-fluid">
                    <div class="span12" id="well-this">
                        @statusTitleMap[Response.StatusCode]
                    </div>
                </div>
        
                <div class="row-fluid">
                    <div class="span11" id="message">
                    @switch (Response.StatusCode)
                    {
                        case 404:
                            @:We cannot find the page you are looking for. If you typed in the address, double check the spelling. If you got here by clicking a link, 
                            @: <a href="mailto:customerservice@domain.com?subject=Page%20Not%20Found">let us know</a>.
                            break;
                        case 410:
                            @:The page you are looking for is gone (permanently). If you feel you reached this page incorrectly, <a href="mailto:customerservice@domain.com?subject=Link%20Gone">let us know</a>.
                            break;
                        case 500:
                            @:Oh dear, something's gone wrong. Our team has already been alerted to the problem and will fix it as soon as possible! 
                            break;
                    }  
                    </div>
                </div>
            </div>
        </div>
    </div>
}

Testing Errors

At the bottom of the home page are some links to test out error functionality.





Handling Not Found and Gone Permanently

404s have some special handling in Global.asax.cs:
protected void Application_EndRequest()
{
	if (Context.Response.StatusCode == 404)
	{
		if (Request.RequestContext.RouteData.Values["fake404"] == null)
		{
			Response.Clear();

			var rd = new RouteData();
			rd.Values["controller"] = MVC.Error.Name;
			rd.Values["action"] = MVC.Error.ActionNames.NotFound;

			var c = (IController)DependencyResolver.Current.GetService<ErrorController>();
			Request.RequestContext.RouteData = rd;
			c.Execute(new RequestContext(new HttpContextWrapper(Context), rd));
		}
	}
}
That code makes me itch a bit and could be done better but there is a reason for it. If you're wondering why not just use custom error pages, the answer is that doing so for a 404 page ends up producing a 301 redirect to then 404 on the custom not found page. For SEO purposes that is usually considered a bad practice. If you don't have a public site or as much SEO concern, that may be acceptable but in this case it wasn't.

Error Controller

The error controller sets and logs the response status codes and returns the error view. For the Gone action, usually there'd be routes defined for legacy URLs that would direct to that action.
public partial class ErrorController : Controller
{
	private readonly ILog _log;

	public ErrorController(ILog log)
	{
		_log = Ensure.That(log, "log").IsNotNull().Value;
	}

	public virtual ActionResult NotFound()
	{
		Response.StatusCode = (int)HttpStatusCode.NotFound;
		RouteData.Values["fake404"] = true;
		_log.Write(LogType.Warn, new { Code = "404" }, 
			"404 Not Found for {0}", Request.Url);
		return View("Error");
	}

	public virtual ActionResult Gone()
	{
		Response.StatusCode = (int)HttpStatusCode.Gone;
		Response.Status = "410 Gone";
		Response.TrySkipIisCustomErrors = true;
		_log.Write(LogType.Warn, new {Code = "410"}, 
			"410 gone permanently for {0}", Request.Url);
		return View("Error");
	}
}

Future Enhancements

  • The diagnostic info report building classes are pretty quick and messy in spots and could use cleanup.
  • Error view work including better responsive design and user-acceptable images :)
  • Admin error view to inspect log data and errors
  • Use and integration of apps such as appfail.net and New Relic to monitor errors and performance. This app used those tools in conjunction with the custom error handling and logging functionality.

The Code

The code for this series is available at https://github.com/thnk2wn/NLogSql.Web
Monday
May202013

ASP.NET NLog Sql Server Logging and Error Handling Part 1

It was the eleventh hour before a web app was to go live and there was nothing in place for error handling or logging. You're shocked I'm sure (that's my sarcastic voice). Normally with web apps in the past I've used ELMAH for unhandled error logging and NLog with text file targets for other logging.

ELMAH works great but in this case I was told it was pulled for some reason. With no time to find out the story there I threw together something to capture basic error details and send an email. Later that grew and was customized and ELMAH never returned though perhaps it should have. Still there are benefits of doing this yourself including more customization, less bloat, less hunting through perhaps lacking documentation etc.

On the logging front I still wanted to use NLog but text file logging was no longer a fit. This app had a few instances behind a load balancer and central logging was needed plus it needed a higher volume of logging in some troublesome workflows, more so than the occasional errors or warnings here and there. I had not used NLog with SQL Server before but this seemed like the perfect time to do so.

Series Overview

Part 1 - Setting up logging with ASP.NET MVC, NLog and SQL Server

Part 2 - Unhandled exception processing, building an error report, emailing errors, and custom error pages.

Setup and Configuration

I began by installing the NLog NuGet package and the NLog.Extended package for asp.net specific information into the ASP.NET MVC 4 project. The NLog.Config package is useful for a standalone NLog config file but for web apps with existing config transforms I find embedding the configuration in Web.config easier.

In web.config under configuration/configSections, the NLog section is defined:
	<section name="nlog" type="NLog.Config.ConfigSectionHandler, NLog" />
Under <configuration> my initial NLog config looked like the below, using the async attribute on targets to make each target work asynchronously:
<nlog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" throwExceptions="true" 
    internalLogLevel="Debug" internalLogFile="${basedir}/NlogInternal.log">
    <!-- go to http://nlog-project.org/wiki/Configuration_file for more information -->
    <extensions>
      <add assembly="NLog.Extended" />
    </extensions>
    <targets async="true">
      <target xsi:type="Database" name="dbTarget" connectionStringName="Logger" 
	  commandText="exec sLogEvent_Insert @time_stamp, @level, @logger, @userName, @url, 
	    @machineName, @sessionId, @threadId, @referrer, @userAgent, @code, @message">
        <parameter name="@time_stamp" layout="${date}" />
        <parameter name="@level" layout="${level}" />
        <parameter name="@logger" layout="${logger}" />
        <parameter name="@userName" layout="${identity}" />
        <parameter name="@url" layout="${aspnet-request:serverVariable=Url}" />
        <parameter name="@machineName" layout="${machinename}" />
        <parameter name="@sessionId" layout="${aspnet-sessionid}" />
        <parameter name="@threadId" layout="${threadid}" />
        <parameter name="@referrer" 
			layout="${aspnet-request:serverVariable=HTTP_REFERER}" />
        <parameter name="@userAgent" 
			layout="${aspnet-request:serverVariable=HTTP_USER_AGENT}" />
		<parameter name="@code" layout="${event-context:item=Code}" />
        <parameter name="@message" layout="${message}" />
      </target>
      <target name="debugTarget" xsi:type="Debugger" 
		layout="${time}|${level:uppercase=true}|${logger}|${message}" />
    </targets>
    <rules>
      <!-- Levels: Off, Trace, Debug, Info, Warn, Error, Fatal -->
      <logger name="*" minlevel="Trace" writeTo="debugTarget,dbTarget" />
    </rules>
</nlog>
Later I swapped to explicit use of AsyncWrapper targets as it allowed me to control the behavior more explicitly and mirrored the reality at runtime, at the cost of being more verbose.
<nlog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" throwExceptions="true" 
    internalLogLevel="Debug" internalLogFile="${basedir}/NlogInternal.log">
    <!-- go to http://nlog-project.org/wiki/Configuration_file for more information -->
    <extensions>
      <add assembly="NLog.Extended" />
    </extensions>
    <targets>
      <target name="asyncDbWrapperTarget" xsi:type="AsyncWrapper" queueLimit="10000" 
	  timeToSleepBetweenBatches="50" batchSize="100" overflowAction="Block">
        <target xsi:type="Database" name="dbTarget" connectionStringName="Logger" 
		commandText="exec sLogEvent_Insert @time_stamp, @level, @logger, @userName, @url, 
		@machineName, @sessionId, @threadId, @referrer, @userAgent, @code, @message">
          <parameter name="@time_stamp" layout="${date}" />
          <parameter name="@level" layout="${level}" />
          <parameter name="@logger" layout="${logger}" />
          <parameter name="@userName" layout="${identity}" />
          <parameter name="@url" layout="${aspnet-request:serverVariable=Url}" />
          <parameter name="@machineName" layout="${machinename}" />
          <parameter name="@sessionId" layout="${aspnet-sessionid}" />
          <parameter name="@threadId" layout="${threadid}" />
          <parameter name="@referrer" 
			layout="${aspnet-request:serverVariable=HTTP_REFERER}" />
          <parameter name="@userAgent" 
			layout="${aspnet-request:serverVariable=HTTP_USER_AGENT}" />
          <parameter name="@code" layout="${event-context:item=Code}" />
          <parameter name="@message" layout="${message}" />
        </target>
      </target>
      <target name="asyncDebugWrapperTarget" xsi:type="AsyncWrapper" queueLimit="10000" 
	  timeToSleepBetweenBatches="50" batchSize="100" overflowAction="Block">
        <target name="debugTarget" xsi:type="Debugger" 
			layout="${time}|${level:uppercase=true}|${logger}|${message}" />
      </target>
    </targets>
    <rules>
      <!-- Levels: Off, Trace, Debug, Info, Warn, Error, Fatal -->
      <logger name="*" minlevel="Trace" 
		writeTo="asyncDebugWrapperTarget,asyncDbWrapperTarget" />
    </rules>
 </nlog>
The database target's connectionStringName value of Logger points to the DB connection string to use.
<connectionStrings>
<add name="Logger" providerName="System.Data.SqlClient" 
   connectionString="Data Source=(local);Initial Catalog=Chinook.Logs;Persist Security Info=True;User ID=ChinookLogger;Password=L0gger!" />
</connectionStrings>

The Database

Our DBA setup a separate database for logging data to set DB properties differently as log data and transactional app data are different beasts. The sample app for this post is written against the Chinook sample database so I named the log database Chinook.Logs. The primary table script follows, minus the indexes, SET statements and the like.
CREATE TABLE [dbo].[LogEvent](
	[LogId] [int] IDENTITY(1,1) NOT NULL,
	[LogDate] [datetime] NOT NULL,
	[EventLevel] [nvarchar](50) NOT NULL,
	[LoggerName] [nvarchar](500) NOT NULL,
	[UserName] [nvarchar](50) NULL,
	[Url] [nvarchar](1024) NULL,
	[MachineName] [nvarchar](100) NOT NULL,
	[SessionId] [nvarchar](100) NULL,
	[ThreadId] [int] NULL,
	[Referrer] [nvarchar](1024) NULL,
	[UserAgent] [nvarchar](500) NULL,
	[Code] [nvarchar](10) NULL,
	[LogMessage] [nvarchar](max) NOT NULL,
	[PartitionKey] [tinyint] NOT NULL,
PRIMARY KEY CLUSTERED 
(
	[LogId] ASC,
	[PartitionKey] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)
The table is pretty straightforward with perhaps a couple of exceptions. The Code field was to be used for custom error codes (HTTP or otherwise) or "tagging" log statements. PartitionKey was added by our DBA mainly for archiving log data but also for efficient querying by day; more on this to come. Finally I originally had the string columns as varchar but we found that NLog used nvarchar and a conversion was happening with each insert that had a slight performance impact.

The insert sproc grabs the day of the week and inserts that as the partition key and the rest of the data just passes through.
CREATE PROCEDURE [dbo].[sLogEvent_Insert]
	@time_stamp datetime,
	@level nvarchar(50),
	@logger nvarchar(500),
	@userName nvarchar(50),
	@url nvarchar(1024),
	@machineName nvarchar(100),
	@sessionId nvarchar(100),
	@threadId int,
	@referrer nvarchar(1024),
	@userAgent nvarchar(500),
	@code nvarchar(10),
	@message nvarchar(max)
AS
BEGIN
	SET NOCOUNT ON;
	Declare @currentDate Datetime
	declare @partitionKey tinyint
	
	set		@currentDate = getdate()
	set		@partitionKey = DATEPART(weekday, @currentDate)

	INSERT INTO [dbo].[LogEvent]
           ([LogDate]
           ,[EventLevel]
           ,[LoggerName]
           ,[UserName]
           ,[Url]
           ,[MachineName]
           ,[SessionId]
           ,[ThreadId]
           ,[Referrer]
           ,[UserAgent]
		   ,[Code]
           ,[LogMessage]
		   ,[PartitionKey])
     VALUES
           (@time_stamp
           ,@level
           ,@logger
           ,@userName
           ,@url
           ,@machineName
           ,@sessionId
           ,@threadId
           ,@referrer
           ,@userAgent
		   ,@code
           ,@message
		   ,@partitionKey);
END
An identical log table was created but with the name LogEvent_Switched. The cleanup sproc was created by our DBA:
CREATE PROCEDURE [DBA].[WeekdayPartitionCleanup_PartitionSwitching] 
AS
BEGIN
	SET NOCOUNT ON;
	declare @partitionKey int
	declare @SQLCommand nvarchar(1024)
	truncate table [Chinook.Logs].dbo.LogEvent_Switched;

	set @partitionKey = datePart(weekday, getdate()) + 1
	if(@partitionkey >7)
		set @partitionKey = 1

	set @SQLCommand = 'alter table [Chinook.Logs].dbo.LogEvent switch partition  ' 
	  + cast(@partitionKey as varchar) + ' to [Chinook.Logs].dbo.LogEvent_Switched;'
	exec sp_executesql @SQLCommand
END
That was scheduled to run daily. So if today is Friday, datePart(weekday, getdate()) returns 6, +1 is Saturday so all of last Saturday's log records get automagically switched over to the LogEvent_Switched table. This leaves the last 6 days of log records in LogEvent and the 7th day in LogEvent_Switched. If you don't have the Enterprise edition and can't use partitioning, regular deletes may work but could be problematic when deleting a large number of rows during frequent inserts or selects.

Finally permission to execute the sproc was granted and then it was back to .net land.
GRANT EXECUTE on dbo.sLogEvent_Insert to ChinookLogger;

Wiring up the Logging Code

First an interface that mostly mirrors NLog's Logger class.
namespace NLogSql.Web.Infrastructure.Diagnostics.Logging
{
    public interface ILog
    {
        void Debug(string format, params object[] args);
        void Error(string format, params object[] args);
        void Fatal(string format, params object[] args);
        void Info(string format, params object[] args);
        void Trace(string format, params object[] args);
        void Warn(string format, params object[] args);

        // custom
        void Write(LogType type, object properties, string message, params object[] args);

        bool IsDebugEnabled { get; }
        bool IsErrorEnabled { get; }
        bool IsFatalEnabled { get; }
        bool IsInfoEnabled { get; }
        bool IsTraceEnabled { get; }
        bool IsWarnEnabled { get; }
    }
}
The functionality of the Log class is provided mostly by inheriting from NLog's Logger class. One custom method is defined to allow passing custom log properties much like you would in ASP.NET (new {code = "404"}).
using System.ComponentModel;
using System.Globalization;
using NLog;

namespace NLogSql.Web.Infrastructure.Diagnostics.Logging
{
    public class Log : Logger, ILog
    { 
        public void Write(LogType type, object properties, string message, 
			params object[] args)
        {
            var info = new LogEventInfo(LogLevel.FromOrdinal((int)type), Name, 
				CultureInfo.CurrentCulture, message, args);

            if (null != properties)
            {
                foreach (PropertyDescriptor propertyDescriptor 
					in TypeDescriptor.GetProperties(properties))
                {
                    var value = propertyDescriptor.GetValue(properties);
                    info.Properties[propertyDescriptor.Name] = value;
                }
            }

            Log(info);
        }
    }

    public enum LogType
    {
        Trace, Debug, Info, Warn, Error, Fatal
    }
}
A log instance is typically created via injecting ILog into the constructor of the class needing logging. Here I use Ninject to wire this up.
using System;
using System.Linq;
using System.Web.Mvc;
using NLog;
using NLogSql.Web.Infrastructure.Diagnostics.Logging;
using NLogSql.Web.Infrastructure.ErrorHandling;
using Ninject.Activation;
using Ninject.Modules;
using Ninject.Web.Mvc.FilterBindingSyntax;

namespace NLogSql.Web.DI
{
    public class DiagnosticsModule : NinjectModule
    {
        public override void Load()
        {
            Bind<ILog>().ToMethod(CreateLog);
            Bind<IErrorReporter>().To<ErrorReporter>();
            Bind<IEventLogWriter>().To<EventLogWriter>();
            Bind<IErrorEmailer>().To<ErrorEmailer>();

            Kernel.BindFilter<AppErrorHandlerAttribute>(FilterScope.Controller, 0);
        }

        private static ILog CreateLog(IContext ctx)
        {
            var p = ctx.Parameters.FirstOrDefault(x => 
				x.Name == LogConstants.LoggerNameParam);
            var loggerName = (null != p) ? p.GetValue(ctx, null).ToString() : null;

            if (string.IsNullOrWhiteSpace(loggerName))
            {
                if (null == ctx.Request.ParentRequest)
                {
                    throw new NullReferenceException(
                        "ParentRequest is null; unable to determine logger name; " 
						+ "if not injecting into a ctor a parameter for the "
						+ "logger name must be provided");
                }

                var service = ctx.Request.ParentRequest.Service;
                loggerName = service.FullName;
            }

            var log = (ILog)LogManager.GetLogger(loggerName, typeof(Log));
            return log;
        }
    }
}
In some cases we can't easily do ctor injection - for example in the application class of Global.asax.cs or in a filter attribute. For these exception cases AppLogFactory is used to create an instance of ILog.
using System;
using System.Diagnostics;
using System.Web.Mvc;
using Ninject;
using Ninject.Parameters;

namespace NLogSql.Web.Infrastructure.Diagnostics.Logging
{
    /// <summary>
    /// Creates a log object for those instances where one cannot be injected (i.e. app startup).
    /// Generally you should just ctor inject ILog
    /// </summary>
    public static class AppLogFactory
    {
        public static ILog Create()
        {
            var declaringType = new StackTrace(1, false).GetFrame(1)
				.GetMethod().DeclaringType;

            if (declaringType != null)
            {
                var loggerName = declaringType.FullName;
                return Create(loggerName);
            }

            throw new InvalidOperationException(
				"Could not determine declaring type; specify logger name explicitly");
        }

        public static ILog Create<T>()
        {
            return Create(typeof(T).FullName);
        }

        public static ILog Create(string loggerName)
        {
            var log = Kernel.Get<ILog>(
				new ConstructorArgument(LogConstants.LoggerNameParam, loggerName));
            return log;
        }

        private static IKernel Kernel
        {
            get
            {
                return DependencyResolver.Current.GetService<IKernel>();
            }
        }
    }

    public class LogConstants
    {
        public const string LoggerNameParam = "loggerName";
    }
}

Exercising the Log

First a simple test, injecting ILog into a controllor ctor, ensuring it isn't null using the Ensure.That NuGet package and logging a count around a database fetch.
public partial class HomeController : Controller
{
	private readonly IMappingService _mappingService;
	private readonly ILog _log;
	private readonly IMusicService _musicService;

	public HomeController(IMusicService musicService, ILog log, IMappingService mappingService)
	{
		_mappingService = Ensure.That(mappingService, "mappingService").IsNotNull().Value;
		_log = Ensure.That(log, "log").IsNotNull().Value;
		_musicService = Ensure.That(musicService, "musicService").IsNotNull().Value;
	}
	
	public virtual ActionResult Index()
	{
		_log.Debug("Retrieving genres");
		var genres = GetGenres().Result;
		_log.Info("Retrieved {0} music genres from the database", genres.Count);
		// ...
	}
}
To get a little more value, an action filter to log each controller action executed along with the execution time.
public class ActionTrackerAttribute : ActionFilterAttribute
{
	private Stopwatch Watch { get; set; }
	private ILog Log { get; set; }
	private ActionExecutingContext FilterContext { get; set; }

	private string ActionName
	{
		get { return FilterContext.ActionDescriptor.ActionName; }
	}

	private string ControllerName
	{
		get { return FilterContext.ActionDescriptor.ControllerDescriptor.ControllerName; }
	}

	private Uri Url
	{
		get { return FilterContext.RequestContext.HttpContext.Request.Url; }
	}

	public override void OnActionExecuting(ActionExecutingContext filterContext)
	{
		base.OnActionExecuting(filterContext);

		try
		{
			FilterContext = filterContext;
			Log = AppLogFactory.Create<ActionTrackerAttribute>();
			Log.Trace("Executing {0}.{1}", ControllerName, ActionName);
			Watch = Stopwatch.StartNew();
		}
		catch (Exception ex)
		{
			Trace.WriteLine(ex);
		}
	}

	public override void OnResultExecuted(ResultExecutedContext filterContext)
	{
		base.OnResultExecuted(filterContext);

		try
		{
			Watch.Stop();
			Log.Info("Executed {0}.{1} for {2} in {3:##0.000} second(s)", 
				ControllerName, ActionName, Url, Watch.Elapsed.TotalSeconds);
		}
		catch (Exception ex)
		{
			Trace.WriteLine(ex);
		}
	}
}

Troubleshooting Failed Logging

When setting up logging initially or making changes later, logging might not work. Usually I'd first fire up SQL Server Profiler and watch for calls to the log insert sproc. If I didn't see them then I'd know it's probably a configuration issue. In the nlog tag in the config file I set throwExceptions="true" internalLogLevel="Debug" internalLogFile="${basedir}/NlogInternal.log" but in practice that rarely seemed to do anything; turning off async temporarily may further help troubleshoot.

First I'd check when creating the log if the log levels were enabled as expected from the config.


Next I'd check the configuration in code with a small class, also useful for changing config at runtime.
public class LogManagement
{
	public static AsyncTargetWrapper GetAsyncDbWrapperTarget()
	{
		var target = (AsyncTargetWrapper)LogManager.Configuration
			.FindTargetByName("asyncDbWrapperTarget");
		return target;
	}
}
From there I'd further inspect the configuration to see if everything looks okay.


If I did see the insert sproc call in SQL Server Profiler then I'd check DB setup and permissions and grab the SQL from Profiler...


... and paste into a query window, then format the Sql (I use Poor Man's TSql Formatter) and execute to check for errors.


Evaluating Async Logging

Because I'm paranoid, OCD, detailed and curious I wanted to ensure the async behavior and compare to non-async. On the home page I added a link to generate batch log records, corresponding to this controller action.
public virtual ActionResult BatchTest()
{
	var sw = Stopwatch.StartNew();
	const int count = 10000;
	for (var i = 0; i < count; i++)
	{
		_log.Trace("Testing {0}", i);
	}
	sw.Stop();
	var msg = string.Format("{0} log invokes in {1:##0.000} seconds", count, 
		sw.Elapsed.TotalSeconds);
	_log.Info(msg);

	return new ContentResult { Content = msg };
}
The async result:


And the non-async result:


Evaluating Log Queries

Usually when inspecting log records I'd select just those columns I'm interested in and order by LogId DESC.


Once enough log data was there it was helpful to query on the columns that would typically be filtered on and evaluate the execution plan and response times. From there indexes were added as needed for columns such as EventLevel, URL, UserName, etc. and execution times compared afterwards.

Preparing for Deployment

In Web.Config.Release NLog config transformations are performed to change the log level from Trace to Info, to remove debugger output, and to tone down NLog internal issue reporting.
<nlog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xdt:Transform="SetAttributes"
        throwExceptions="false" internalLogLevel="Warn" 
		internalLogFile="${basedir}/NlogInternal.log">
    <rules>
       <!-- Levels: Off, Trace, Debug, Info, Warn, Error, Fatal -->
       <logger name="*" minlevel="Info" writeTo="asyncDbWrapperTarget" 
	      xdt:Transform="SetAttributes" xdt:Locator="Match(name)" />
    </rules>
</nlog>
Slow Cheetah preview:
Note that removing the nlog xsd attribute in the root web.config (and all transforms) was needed for the transformation to work correctly. Also at one point I tried a replace transform to replace the entire nlog tag but found it didn't work correctly and it created too much duplicate content between configurations.

Some Queries


Everything today:
SELECT *
FROM LogEvent WITH (NOLOCK)
WHERE PartitionKey = (
		SELECT DATEPART(weekday, getdate())
		)
ORDER BY logid DESC;
Errors and warnings in the past 2 hours:
SELECT *
FROM LogEvent WITH (NOLOCK)
WHERE eventlevel IN (
		'Error'
		,'Warn'
		)
	AND DATEDIFF(hh, LogDate, GETDATE()) <= 2
ORDER BY logid DESC;
Errors by day and url:
SELECT cast(logdate AS DATE) [Day]
	,url
	,count(*) ErrorCount
FROM LogEvent WITH (NOLOCK)
WHERE eventlevel = 'Error'
	AND len(url) > 0
GROUP BY cast(logdate AS DATE)
	,url
ORDER BY Day DESC
	,Url;
Errors by logger name / class name / component area:
SELECT LoggerName
	,count(*) AS ErrorCount
FROM LogEvent WITH (NOLOCK)
WHERE EventLevel = 'Error'
GROUP BY LoggerName
ORDER BY count(*) DESC;
Unhandled exceptions today:
SELECT *
FROM LogEvent WITH (NOLOCK)
WHERE PartitionKey = (
		SELECT DATEPART(weekday, getdate())
		)
	AND eventlevel = 'Error'
	AND LoggerName IN ('NLogSql.Web.Infrastructure.ErrorHandling.IErrorReporter')
ORDER BY logid DESC;

Future Enhancements / Considerations

  • Web view to read, search, filter log maybe with some SignalR goodness
  • AppId type column if logging from multiple apps w/o desire for multiple log databases
  • Possibly a separate error table that the log table could link to for errors with the full error report split up

The Code

The code for this series is available at https://github.com/thnk2wn/NLogSql.Web
Monday
May142012

Build Automation Part 4: Database and Report Deployments

Series Index

Build Automation Part 1: Overview and Pre-build Tasks
Build Automation Part 2: Building and Packaging
Build Automation Part 3: App Deployment Script
Build Automation Part 4: Database and Report Deployments

Unlike deploying the application itself in its entirety each time, database and report items have required incremental deployments due to their nature and sheer size. The question becomes how to manage these increments in coordination with the application bits to ensure everything is in sync.

I do not pretend to have all the answers here. I was hoping this part of the process could get overhauled more with our last release but there is only so much time in the day.

Some Challenges

Size, Dependencies, Impact Analysis

On the (Oracle) database side of our operational app we have around 800 tables, 500+ packages, 500+ views, 300+ functions and procedures, and a variety of other object types across 3 schemas. Running schema compares / diffs and doing sync scripts has been a time consuming pain in the past, regardless of various tools we have tried.

The situation is complicated by some 10 database environments, other app databases our app is dependent on, various apps depending on our database, and dynamic SQL embedded into report RDL files or coming from ORM's like Entity Framework. Dependency and impact analysis can be difficult, particularly across systems.

On the report side this app has over 600 SSRS reports scattered over 8 servers, several folders and different versions of SQL Server Reporting Services.

Source Control

Source control for reports has not been a problem so much with using SSRS, other than TFS generating a lot of unnecessary merge conflicts on those XML RDL files.

On the database side we have had some success using Team Coding in Toad with the TFS MSSCCI Provider. Quest came out with a native TFS provider but it did not support TFS work item association which ruled out our use of it.

The MSSCCI provider "works" with Toad for basic changes like packages, procedures, functions, triggers, views, etc. but boy it is not without a lot of quirks. So much so that for a while I saved any database object script changes to files on my desktop out of fear my changes would get overwritten, which used to happen quite a bit. The other problem is that not all changes are source controlled such as table schema changes, data changes etc. We offset that via some SQL file attachments to tasks tied to a project.



Pulling Report and Database Changes



I posted before about a custom TFS Artifact Manager (hereinafter "TAM") tool for pulling down report and database changeset files and task attachments either for a single work item or a large batch of related work items, such as all tasks linked through a series of scenarios, backlog items and bugs tied to a project. I won't repeat those details here but we currently still use the tool to gather up all the report and database changes for a given project release. It far from perfect but it beats manual guesswork and building sync scripts from database compares.

The TAM tool is also used to pull artifacts from various single task ad-hoc changes made outside of any official product release. Many reports can be added to the application dynamically through some dynamic report forms with common parameters; new records are added to the database and the reports are deployed outside of any app deployment. Likewise there are occasional database changes made independent of the application.

There are other tools mentioned in the TAM post that we may try using more in the future. Also, Troy Hunt has a good series of deployment posts including this one regarding Automated database releases with TeamCity and Red Gate. Doing exactly that with our schemas would make me a bit nervous but perhaps with tweaks and some experiments in the future.

Additionally I posted a review on Red Gate's Schema Compare for Oracle which can be quite a useful tool. We don't rely on it as heavily anymore with custom tools and processes but it is handy to use it to double-check things after deployments or for doing more one-off database syncs.

Deploying Reports

Ad hoc Report Deployments

When deploying from Development to Test, our CI process picks up any reports merged from $/MyApp/Reports/Dev/ to $/MyApp/Reports/Main/ and automatically deploys those via PowerShell and RS Utility. Any corresponding database changes are manually applied before that. Environments beyond Test currently require attaching report and database changes to a Help Desk ticket and routing to a DBA. The DBA runs any database changes and deploys the reports using the script mentioned in the next section.

App Report Deployments

For reports to be deployed with a given build of an application, the process is basically the same as the ad-hoc process in respect to going from Dev to Test. One difference is on timing of merging the report changes in source control to correspond with any dependent changes to the application code. When moving beyond the Test environment, all reports tied to a given project work item are pulled using the TAM tool. They are then deployed in mass using PowerShell and the SSRS web service (without RS Utility), in a manner similar to this post on Uploading SSRS Reports with PowerShell.



Deploying Database Changes

We currently do not have much automation around ad hoc database deployments but not much is needed there.

For app database changes we start by pulling the database changes using the TAM tool. In that tool a script "package" is built by choosing the scripts to include and specifying any needed order or execution. Previously someone (dev or DBA depending on environment) would either execute all those manually by hand in Toad, or build out an index/driver script and run that. It was not as bad as it might sound, given the tool produced combined SQL scripts for views, packages, procs, etc. Still it was tedious if there were a number of data or schema migration scripts to be run in order.

Executing Database Scripts With PowerShell

Our resident Oracle expert Jim Tilson ("The Optimizer") had the idea of creating a PowerShell script that used SQL*Plus to generically execute all database scripts in a given directory. I paired with him to get the basic interaction going but this is his brainchild and work. He should probably be the one explaining this but no telling when that slacker will get around to blogging :). If you have any interest in Oracle, SQL optimization, database tech in general, or Ruby, you should reach out to him on Twitter and ask him to blog more (and tweet more while he is at it). At any rate this might be useful for others so I will post the code and attempt to explain it.

Structure

The script expects all the database script files to be located in subfolders where the script resides, one folder per schema name, and no subfolders with each schema folder (not recursive).


Each file in a schema folder will be executed regardless of filename extension. Ordering is based on filename; our TAM tool prefixes a numeric wart on each file to ensure an obvious order. At the moment the script does not explicitly specify a name ordering but that's the default.


Executing Scripts for Each Schema

At the bottom of the script, the server TNS name is prompted for and a couple things are set before running the main run-scripts function.
set-location (Get-ScriptDirectory)
$dateWart = Get-DateWart
$server = read-host "Enter the server TNS name"
run-scripts
echo "Successfully ran all scripts."
Run-Scripts invokes a function to run the scripts for each schema, passing along the user and server info. This could be made more generic by assuming any subfolder where the PowerShell script resides represents a schema with database scripts to run.
function Run-Scripts
{
    foreach ($user in @("USER_SCHEMA_1", "USER_SCHEMA_2", "USER_SCHEMA_3")) 
	{ 
		run-scriptsforschema -server $server -user $user
	}
}

Running Scripts for a Schema

This function will temporarily set location to the schema subfolder corresponding to the user/schema name passed in. It prompts for a password for a later connection to that schema. Finally it enumerates all files in the schema folder, calls a Run-Script function to execute each, and writes progress as it goes.

Each successfully executed file is moved into a Completed subfolder. That file move is important as many migration scripts are written assuming they will only be run once and we have been bit by DBA's accidentally running scripts more than once.
function Run-ScriptsForSchema($user)
{
    echo "Running scripts for $user."
    push-location    
    set-location $user    
	$password = get-password($user)    
    ensure-directory(".\Completed")

    $files = @(get-childitem | where {!$_.PsIsContainer})
	$count = 0
	
    foreach ($fileInfo in $files)
    {        
        write-progress -activity "Running scripts for $user" `
			-currentoperation $fileinfo.name -status ("Executing") `
			-PercentComplete (100*$count/$files.count)
        Run-Script $user $password $fileInfo
        $count++
        write-progress -activity "Running scripts for $user" `
			-currentoperation $fileinfo.name -status ("Done") `
			-PercentComplete (100*$count/$files.count)
        move-item -path $fileInfo.fullname -destination ".\Completed" -Force
    }
    
    write-progress -activity "Running scripts for $user" -status ("Complete") -completed
    pop-location    
    echo "Completed scripts for $user"
}

Running a Script

The Run-Script function takes care of some logging and calls a normalize function to tack on additional SQL before and after the SQL contained in the file (error handling, commits, etc.); more on that in a moment. Function notes follow.
  • Path - the sqlplus location should be in the SYSTEM PATH environment variable so fully qualifying it should not be needed. In my case the location is C:\app\[username]\product\11.2.0\client_1\.
  • -L Parameter - instructs the app to only attempt to log on once; otherwise w/bad credentials it can get hung awaiting input.
  • -M Parameter - indicates HTML output is desired from sqlplus.
  • Credentials - The server name captured earlier is passed in along with the user/schema and password parameter values.
  • SQL - the normalize script function returns a SQL string and that is piped into sqlplus to be executed.
  • Output - the output is sent to $logFile and the 2>$1 sends standard error to standard output.
  • Error checking - Finally $LASTEXITCODE is checked to see what sqlplus.exe exited with; if 0 it was successful, otherwise it is the Oracle error number. The process stops on any error; manual changes might be needed to address any problems then the script can be run again.
function Run-Script($user, $password, $fileInfo)
{
    $logDir = ("..\Logs\{0}\{1}" -f $dateWart, $user)
    ensure-directory $logDir
    $logFile = join-path $logDir ($fileInfo.basename + ".html")
    
    (normalize-script $fileinfo.fullname) | sqlplus.exe -L -M "HTML ON SPOOL ON" `
		-S "$user/""$password""@$server" >> $logfile 2>$1

    $lec = $LASTEXITCODE
    
    if ($lec -ne 0)
    {
        write-error ("ERROR executing {0}!" -f $fileInfo.FullName)
        exit
    }
}

Adjusting the SQL

There are two critical adjustments made to the SQL read from the database script files to execute. The first is detecting a SQL error and exiting SQL*PLus with the error code. The other is issuing a commit at the end; most of our data related scripts do not include a commit as often they are ran and verified before issuing a commit. It is worth reading over the WHENEVER SQLERROR documentation as some types of errors will not trigger an exit; fully checking for all types of errors might require something more brittle like scanning the log files for certain error phrases.
function normalize-script($filename)
{
@"
    whenever sqlerror exit sql.sqlcode
    set echo off
    set termout off
    $([string]::join("`n", (get-content $fileinfo.fullname -readcount 0)))
    commit;
    exit
"@
}

Helper Functions

At the top of the script are some helper functions and an interop services dll is loaded for later use in translating the secure password to a plain text string to be passed along to SQL*Plus.
[Reflection.Assembly]::LoadWithPartialName("System.Runtime.InteropServices")

function Get-ScriptDirectory
{
    Split-Path ((Get-Variable MyInvocation -scope script).Value.MyCommand.Path)
}

function ensure-directory($dir)
{
    if (!(test-path $dir)) { new-item $dir -type directory }
}

function Get-DateWart()
{
    get-date -uformat "%Y %m %d %H %M %S"
}

function get-password($user)
{
    $enterpassword = read-host -AsSecureString "Password for $user@$server"
    [System.Runtime.InteropServices.Marshal]::PtrToStringAuto(`
		[System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($enterpassword));
}

Analyzing the Results

The script stores logs under a Logs\[Timestamp]\Schema\ folder for troubleshooting and verification purposes.


SQL*Plus Alternatives

One alternative to SQL*Plus is using OracleCommand's ExecuteNonQuery method in Oracle.DataAccess.dll. I tried this approach back when I created an OracleScriptExecutor utility app that was designed to easily run SQL Scripts against multiple schemas. It was a bit of a nightmare that I do not recommend. For one you have to deal with annoyances like linefeed issues, semicolon and BEGIN/END block issues, and it is quite difficult to deal with multiple scripts combined in one SQL file (i.e. '/' delimited). It almost requires a full blown SQL parsing engine to handle it correctly so I'd rather delegate that pain to a tool like SQL*Plus that already handles such complexity.

In Conclusion

This really only scratches the surface of the problems and solutions in dealing with database and report deployments. With some time and TLC I am sure this beast could be further tamed. Thoughts, suggestions, tips, helpful tools, processes? Leave a comment!

I am running out of time in this series but hopefully I can touch on some CI details with TeamCity next.