Log management and intelligence

Log Management (LM) comprises an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event-logs, etc). LM covers log collection, centralized aggregation, long-term retention and log analysis (in real-time and in bulk after storage). Syslog offers the most common example of such log messages.

Systems administrators usually perform LM analysis for reasons of security, of operations (such as system or network administration) or of regulatory compliance.

Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection logs).

Users and potential users of LM can build their own log management and intelligence tools, assemble the functionality from various open-source components, or acquire (sub-)systems from commercial vendors.

Deployment life-cycle

One viewFact|date=October 2007 of assessing the maturity of an organization in terms of the deployment of log-management tools might useOr|date=October 2007 successive categories such as:

* Level 1: in the initial stages, organizations use different log-analyzers for analyzing the logs in the devices on the security-perimeter. They aim to identify the patterns of attack on the perimeter infrastructure of the organization.

* Level 2: with increased use of integrated computing, organizations mandate logs to identify the access and usage of confidential data within the security-perimeter.

* Level 3: at the next level of maturity, the log analyzer can track and monitor the performance and availability of systems at the level of the enterprise — especially of those information-assets whose availability organizations regard as vital.

* Level 4: organizations integrate the logs of various business-applications into an enterprise log manager for better value proposition.

* Level 5: organizations merge the physical-access monitoring and the logical-access monitoring into a single view.

List of log management software

** [http://loglogic.com LogLogic Log Data Management & Intelligence]
** [http://www.sensage.com/English/Products/Log_Management.html SenSage Log Warehouse]
** [http://www.splunk.com Splunk IT Index and Search Engine]
** [http://www.logrhythm.com LogRhythm Enterprise Log and Event Management]
** [http://www.prismmicrosys.com EventTracker Enterprise Event Log Management]
** [http://au.geocities.com/bazsyslog1/ BazSyslog]
** [http://www.kiwisyslog.com/ Kiwi Syslog Daemon]
** [http://www.op5.com/op5/products/logserver/ Logserver]
** [http://www.monitorware.com/en/Product/product_comparision.php MonitorWare Products: MonitorWare Agent, WinSyslog]
** [http://www.netmechanica.com/products/?prod_id=1016 NetDecision LogVision]
** [http://ntsyslog.sourceforge.net/ NTsyslog]
** [http://developer.sysco.ch/php/radius_class_pure_php.zip Pure PHP syslog client class]
** [http://www.syslserve.com/ Syslserve]
** [http://www.balabit.com/network-security/syslog-ng/central-syslog-server/ syslog-ng Agent for Windows]
** [http://www.snmpsoft.com/syslogwatcher/ Syslog Watcher]
** [http://www.loriotpro.com/Products/SyslogCollector/SyslogDataSheet_ENv3.php Syslog Collector] A Syslog server/agent for Windows
** [http://tftpd32.jounin.net/ Tftpd32] Tftpd32 which include a syslog server
** [http://www.theonesoftware.com/syslog_manager.php TheOne SysLog Manager]

ee also

*Server log
*Web log analysis software
*Web counter
*Data logging
*Common Log Format
*Syslog

References

* Chris MacKinnon: "LMI In The Enterprise". "Processor" November 18, 2005, Vol.27 Issue 46, page 33. Online at http://www.processor.com/editorial/article.asp?article=articles%2Fp2746%2F09p46%2F09p46.asp, retrieved 2007-09-10

* Mike Rothman: "Looking at Log Management Pragmatically". "EventSource" August, 2007. Online at http://www.prismmicrosys.com/newsletters_august2007.php