As Information Systems (IS) continue to be more and more complex due to greater adoption and sophistication of planned usage, the concerns surrounding the effectiveness of such software projects become more and more critical to the CIO and his team. Further for the IT/ITeS Service Providers, it becomes more critical with the growing understanding that just implementing progressive and ambitious software projects within organizations may not necessarily enhance organization productivity and lead to competitive advantage that cannot be overtaken by the competitors. This is where a deeper understanding of the critical measures of Information Systems Effectiveness comes so that the concerns may be addressed at the grassroot level.So what could be some of the key indicators of information system ineffectiveness? Have you noticed any of these in your organizations?

• Excessive down time and idle time of implemented software projects
• Slow response time of project deliverables
• Excessive maintenance costs of systems
• Unreliable outputs across systems and deduplication of data
• Erroneous data across the different functional departments
• Excessive run costs during system uptime
• User dissatisfaction with output, content or timeliness

The measures or criteria for Information Systems Effectiveness can be broken down hierarchically as highlighted in the following diagram, to address specific dimensions of utility from IS project implementations.

RELATIVE EVALUATION

Task Accomplishment: An effective IS project is expected to improve the task accomplishment of the system users. However it is important to note that providing specific measures of past accomplishment that auditors can use for evaluation can be difficult. Performance measures for task accomplishment differ across applications and sometimes across organizations. For example, for a manufacturing control system might be number of units output, number of defective units reworked, units scrapped, amount of down time/idle time. Similarly, it is important to trace task accomplishment over time. System may appear to have improved for a short time after implementation, but fall into disarray thereafter.

Quality of Working Life: High quality of working life for users of a system is a major objective in the design process. For example the virtual offices ensure better work-life balance (Accenture, IBM).  However there are variations on the definition & measurement of the concept of quality of working life.  Different groups may have different vested interests : e.g. productivity vs social interest like how to evade work deliverables and engage in social behavior. Some of the major advantages of this measure is that it is relatively objective, verifiable, and difficult to manipulate. Data required is relatively easy to obtain. However some of the major disadvantages of this measure is that it is difficult to link them to IS quality and difficult to pinpoint what corrective action is needed.

ABSOLUTE EVALUATION

1. Operational Effectiveness: The Systems Auditor examines how well an Information system meets its goals from the viewpoint of a user who interacts with the system on a regular basis. Four measures used for estimating operational effectiveness: Frequency of use, Nature of use, Ease of use and User satisfaction. If a system is being frequently used, it is likely to be more effective based on the perspective of its end user. Similarly a very sophisticated system may have analytics and insights dashboards, but it may only be used to record transactions. This lower nature of usage by the end user would indicate lower effectiveness of the implemented information systems. Ease of use and user satisfaction after usage are evaluated from the perspective of the end user. This gives the auditor the additional insight whether there is any productivity loss when the systems are complex and difficult to be handled from the user interface perspective.
2. Technical Effectiveness: In this evaluation the focus of the auditor is to evaluate whether the implemented Information System has the appropriate hardware / software technology been used to support a system, or, whether a change in the technology would enable the system to meet its goals better. Hardware performance can be measured using hardware monitors or more gross measures such as system response time, down time. Software effectiveness can be measured by examining the history of program maintenance, modification and run time resource consumption. The history of a software or hardware repair maintenance indicates the quality of logic existing in a program. Extensive error correction implies on a frequent basis implies inappropriate design, coding or testing; failure to use structured approaches, etc. How ever there is a major challenge in using these measures. One needs to understand that hardware & software are not independent resources and the effectiveness audit needs to account for possible overlaps.
3. Economic Effectiveness: This requires the identification of costs and benefits and the proper evaluation of costs and benefits. Sometimes, achieving it is difficult since costs and benefits depend on the nature of the Information Systems Project. Not all costs and benefits can be tracked directly in many business functions. Some of the benefits expected and derived from an IS would be based on the contextual usage. For example, a system designed to support a social service environment vis-à-vis a system designed to support manufacturing activities would have different outcomes. Some of the most significant costs and benefits may be intangible and difficult to identify, and next to impossible to value. So an auditor needs to be careful while attempting measures to evaluate a system based on these criteria.

Further there are two different contrasting approaches for evaluating information system effectiveness.

• Goal focussed view – In this view, the evaluation focuses on whether the planned information system actually achieve goals set out in the initial planning phase? Conflicts as to priorities, timing etc. can lead to objectives met in the short run by sacrificing fundamental system qualities, leading to long run decline of effectiveness of the system
• System resource view – In this view, the desirable qualities of a system are identified and their levels are measured by the system auditors. If the qualities exist, then information system objectives, by inference, should be met. By measuring the qualities of the system may get a better, longer-term view of a system’s effectiveness.
• Major challenge in such approaches: measuring system qualities is much more difficult than measuring goal achievement.

It is important to note that beyond these effectiveness measures, there are other dimensions that auditors may need to account for while conducting an information systems audit. Efficiency audits are equally important for information systems projects and a detailed understanding may be required while conducting them thoroughly.

All these criteria and measures collectively can be used to evaluate the effectiveness of any software project / information systems project. Conducting such audits on a regular basis becomes critical for the success of any planned software deployment.

Please feel free to write to us if you need assistance in conducting an Information Systems Project Audit. We would love to conduct an audit of the software projects implemented in your organization.

• Technology