Shadow system

Shadow System is a term used in Information Services for any application relied upon for business processes that is not under the jurisdiction of a centralized Information Systems department. That is, the Information Systems department did not create it, was not aware of it, and does not support it.

Overview

Shadow Systems (aka Shadow Data Systems, Data Shadow Systems, Shadow Information Technology or in short: Shadow IT) consist of small scale databases and/or spreadsheets developed and used by end users, outside the direct control of an organisation's IT department.

Typically these systems are developed on an adhoc basis rather than as part of a formal project and are not tested, documented or secured with the same rigour as more formally engineered reporting solutions. This makes them comparatively quick and cheap to develop, but unsuitable in most cases for long term use.

The term can also refer to legitimate, managed replicas of operational databases that are isolated from the user base of the main system. These sub-systems can be used to track illegitimate changes to the primary data-store by 'back doors' exploited by expert but un-authorized users.

As stated in Price Waterhouse Coopers report on Spreadsheet Risk Management “The Use of Spreadsheets:Considerations for Section 404 of the Sarbanes-Oxley Act” :

“Many companies rely on spreadsheets as a key component in their financial reporting and operational processes. However, it is clear that the flexibility of spreadsheets has sometimes come at a cost. It is important that management identify where control breakdowns could lead to potential material misstatements and that controls for significant spreadsheets be documented, evaluated and tested. And, perhaps more importantly, management should evaluate whether it is possible to implement adequate controls over significant spreadsheets to sufficiently mitigate this risk, or if spreadsheets related to significant accounts or with higher complexity should be migrated to an application system with a more formalized information technology control environment.”

Cause

An organization that has a centralized Information Services department usually requires rigorous guidelines for developing a new system or application. Simultaneously, with the rise of powerful desktop applications that give savvy end-users the ability to author sophisticated tools on their own, a business group often finds it more expedient to create the application themselves.

Pressure to analyse information in new ways

Any organisation faces a multitude of pressures to change and respond to new government regulations, customer demands and action by competitors. In order to respond to these changes, organisations need to be able to understand all aspects of their business and often ask questions of itself that have never been asked before.

Ongoing pressure for change creates an ongoing pressure to analyse data in new ways and get information quickly into the hands of people who need it. Only through creative and flexible reporting are businesses able to spot new trends and identify new opportunities rapidly enough to take full advantage of them.

Increased power of personal computer hardware and software

The greatly increased power of personal computer hardware and software analysis tools has meant that individual users now have all of the computing power they need right in front of them. Large databases containing all of an organisation's customer or supplier information; the kind that could once only be stored on a central corporate mainframe, can now be contained easily on a single laptop.

Rigorous controls and the breadth of required skills leads to unresponsive IT departments

Quite properly, when a reporting system is put together by IT professionals, they need to consider all aspects of how the system will be used. In addition to just putting the information together they need to consider.

* How can it be ensured that the data produced is accurate?
* Who is authorised to see this information? How can security be enforced?
* How is the system to be backed up/replicated in case of failure?
* User documentation must be written so that the system can be given to new users.
* Technical documentation must be produced so that support staff can maintain it.
* The load that any new tool places on existing systems needs to be managed and minimised.

The various skills that are required to achieve all of this means that inevitably a number of different people will all be involved in the task of creating the new report. This increases the amount of time and effort it takes to put a rigorously engineered solution in place. Shadow Systems typically ignore this kind of rigor, making them much faster to implement, but less reliable and more difficult to maintain.

Problems

Poorly Designed

Most Shadow Data Systems are created by end users whose main area of expertise is something other than software engineering. Professional developers are trained to develop software in such a way that it can be easily maintained and extended. Because they are written by non-specialists, Shadow Data Systems often suffer from poor design, making errors hard to find and modifications difficult.

Not Scalable

Typically, Shadow Data Systems are only used by one or two people. However useful they are, it’s often impossible to scale them up to support tens or hundreds of users.

Poorly Documented

Shadow Data Systems are often only partially documented if at all. Knowledge about the system is passed by word-of-mouth and can be confined to a very small number of people. This knowledge is then lost completely if one or two staff members leave.

Untested

Around two thirds of the effort involved in professional software development is expended in testing. Shadow Data Systems undergo much more cursory testing and may have latent errors that only become apparent after a long period of production use.

May Allow Unauthorised Access to Sensitive Information

Shadow Data Systems hold substantial chunks of company data and can include confidential information about customers, suppliers or staff. The access control processes for these systems are often much more lax than for a centralised company database and may not even exist at all. Physically locating sensitive data on desktop or laptop computers can leave an organisation very exposed if the computer is stolen.

Easy to Introduce Errors

Data in local databases and spreadsheets can very easily be modified, either intentionally or otherwise. Once changed it can be hard to track what changes have been made and what the original data looked like. Where the system manipulates the data it can introduce more subtle errors that remain completely undetected for long periods.

One Hard Disk Failure away from Disaster

Shadow systems existing on a single computer are often not regularly backed up

Several Versions of the Truth

There may be many different Shadow Systems within an organisation reporting against the same data. Each one may add filters and manipulate the data in different ways. This can lead to apparent inconsistencies in their output. Where two Shadow Systems disagree, either or both of them may be wrong.

References

* [http://www.shadowdatasystems.com Shadow Data Systems]
* [http://www.pwc.com/images/gx/eng/fs/insu/rt5.pdf Price Waterhouse on Spreadsheets Risk Management]