Loggly Optimization for IIS logs

July 19, 2016 Rick ThamMike Skutta

For two years now, we have been using Loggly at One North, and it has proven to be a valuable DevOps tool for our organization.  This service allows us to view and search our logs quickly across environments and applications such as Sitecore and other web CMS applications.

Loggly is a SaaS provider that charges by the amount of data that is being sent to them for processing.  Over time, our cost naturally started to increase as we rolled out Loggly to more servers and applications that we managed.  The bulk of our data consumption to Loggly was our web server logs. 

After having an optimization session with a Loggly technical support engineer, we identified that the web traffic logs were being duplicated to their service: once as JSON and then again as unparsed key value pair data. Up until this point, we ignored the unparsed message and assumed that it was needed for Loggly.

Loggly optimization unparsed message

When we first started using Loggly, we leveraged their source setup examples online to configure our systems to send logs to their service. The recommended approach is to use NXLOG, an open source universal log collector, to send IIS web traffic logs.  NXLOG is a lightweight and efficient service that sends the logs to Loggly. 

After digging into this unparsed data, it was determined that it was not needed by Loggly and that the NXLOG default behavior was sending our data over in two formats. Luckily, we discovered a relatively straightforward fix.

The following are our modifications to the nxlog.config file to remove the unparsed data from the logs as they are sent to Loggly.  The commented example below shows you how to do it.

Once you make this change to NXLOG, you will no longer see the unparsed data within your server logs in Loggly. 

Hopefully, you not only find this post helpful, but end up saving a few bucks, too. 

Subscribe

The One North Ideas Update delivers each month’s latest posts on digital for PSOs—including industry trends, news and our latest research—directly to your inbox. Although it’s our goal to always include thought-provoking and compelling content, you can unsubscribe at any time. 

See our Privacy Policy to learn more about how we protect and manage your submitted data.

Rick Tham Director, Technology

Rick Tham is the Director of Technology at One North, working to research, test and integrate new technologies for both One North and our clients. Rick is the only Rick at our company…and, yes, this was his fun fact.

  • Favorite movie: Ferris Bueller's Day Off. Before I moved to Chicago, this movie was my main source of things to do in Chicago.
  • What you wanted to be when you were little: A doctor, but I guess my patients are now just applications. They get sick too.
Mike Skutta Architect

As an Architect within the Technology Labs group at One North, Mike works to research, test and integrate new technologies for both One North and its clients. He is also a Sitecore Technology MVP – a distinction received by only 217 people worldwide – and holds more than 14 years of experience architecting, designing, developing software and leading technology teams. Based in Florida, Mike is often seen wheeling around the office via our robot.

  • Favorite vending machine snack: Doritos
  • Most unusual job: Working as a Serf at Medieval Times

One North Interactive 
222 North LaSalle St, #1500
Chicago, IL 60601

+1 312.469.1740