Skip to Main Menu

Serilog and the new Splunk Sink

Over the last couple of years, like most in software I have become more and more involved with the “DevOps” landscape. One product that has been a great help when working with the IoT is Splunk. In my opinion, when dealing with distributed systems that utilise many devices and operating systems, the architecture of Splunk simply stacks up. This post will not go into the details of how Splunk works, rather how you can start utilising it in the world of .Net.

Day to day I am primarily in the Microsoft space, and as such .Net is a large part of what I do. In the .Net logging space there have always been the constant libraries (NLog, Log4Net etc.), however with the introduction of libraries such as Serilog the .Net community is better understanding the benefits of structured logging. This is not to say that this was not possible prior to Serilog, however the barrier to entry and understanding is much lower.

##.Net Options for logging to Splunk

There are a number of options out of the box that can get you up and running with Splunk using either a traditional logging provider or something like Serilog.

Index a file

The simplest example of getting up and running is to write to a log file and then use a forwarder or indexer on that file. Check out Configuring Data Inputs on how to set up data inputs in Splunk.

When using Serilog, life is quite simple. Target a rolling file or whatever variant you desire, log the file out to a location that can be accessed by Splunk.


var log = new LoggerConfiguration()

Below is an example output of an indexed file.


##Splunk Sink for Serilog

There is currently a Splunk sink available for Serilog however it is built on top of the old Splunk SDK package (e.g. Install-Package SplunkSDK).

Recently, Glenn Block, Grigori Melnik and the rest of the dev team @ Splunk have re-written the libraries from the ground up.

Get the new awesome sauce Install-Package Splunk.Client and Install-Package Serilog.Sinks.Splunk

NOTE: Currently the examples depend on a prerelease package (e.g. Install-Package Splunk.Client -Pre). This is due to the Splunk PCL package being a beta however it should be generally available sometime this month.

There are a number of options available to an app targeting either PCL or .net 4.5 when using the new Splunk Serilog sink.

  • TCP (not currently available in PCL)
  • UDP (not currently available in PCL)
  • HTTP/HTTPS (available to both full and PCL)

The sinks follow the Serilog convention in logging the rendered message, message template and all enriched properties.

As a result, in Splunk you get something like the following.

``` { "Timestamp":"2014-08-30T09:33:19.6642692+10:00", "Level":"Information", "MessageTemplate":"Logging an int, what fun {Item} for {@person} at {thetime}", "RenderedMessage":"Logging an int, what fun 10 for Person { FirstName: \"Jimmy\", LastName: \"Little\", DateOfBirth: 08/30/1938 09:33:19 } at 08/30/2014 09:33:19", "Properties":{ "Item":10, "person":{ "_typeTag":"Person", "FirstName":"Jimmy", "LastName":"Little", "DateOfBirth":"1938-08-30T09:33:19.4025283+10:00" }, "thetime":"2014-08-30T09:33:19.6642692+10:00", "ThreadId":11 } } ```

So to get going, pull in the new sink

> Install-Package Serilog.Sinks.Splunk -Pre

Install-Package Serilog.Sinks.Splunk


The first thing to do is setup TCP. To do this, check out the steps located in this link.

For my example, I am using 10001 as my listening TCP port within Splunk. From there I need to configure Serilog.

``` var host = ""; var tcpPort = 10001; var log = new LoggerConfiguration() .WriteTo.SplunkViaTcp(host, tcpPort) .CreateLogger(); ```


In a very similar fashion to setting up TCP, you need to configure UDP. The following link will get you started.

In my example, I have used 10000 as my listening UDP port within Splunk. After that, I configure Serilog as below.

``` var host = ""; var udpPort = 10001; var log = new LoggerConfiguration() .WriteTo.SplunkViaUdp(host, udpPort) .CreateLogger(); ```

Below is an example output when logging using TCP. ![Example Logging To TCP](/images/2014/Sep/SplunkAndSerilog_TCP.PNG) ### Via HTTP/HTTPS The final option, and the only one currently available to [Serilog][Serilog] in `PCL` land, is HTTP/HTTPS. This is great for mobile apps and other occasionally connected devices. Generally, the [Splunk][Splunk] team recommend logging via a persistent mechanism (file combined with a forwarder) rather than HTTP/HTTPS. This is normally due to the volume of data being logged. If using a default [Splunk][Splunk] setup, no extra configuration is required. By default the management port of [Splunk][Splunk] is configured to 8089. The following example illustrates how to configure [Serliog][Serilog] to use HTTPS.
``` //This is the Splunk SDK context, specifing the scheme, host and port var host = ""; var port = 8089; var splunkContext = new Context(Scheme.Https, host, port); //Args are used when pushing to Splunk, var transmitterArgs = new TransmitterArgs { transmitterArgs.Source = "My Machine Name", transmitterArgs.SourceType ="MySplunkTest", } //Serilog Context wrapper var serilogContext = new SplunkContext( splunkContext, "mysplunktest", "mySplunkUser", "mySplunkPassword", null, transmitterArgs); var size = 15; var interval = new TimeSpan(0,0,0,5); //Note the batch size and time span Log.Logger = new LoggerConfiguration() .Enrich.With(new ThreadIdEnricher()) .WriteTo.SplunkViaHttp(context, size, interval) .CreateLogger(); ```

What Next

This is just a quick tale indicating how you might get up and running with Splunk. Word is the new Splunk PCL package will be available very soon.

GA is OUT!

Get Amongst It!!!

comments powered by Disqus