Good IT Auditor: Sit, Stay, Understand?

For many companies, the end of the calendar year signals the beginning of the audit season. A review of an organizations information technology is an integral part of this effort. The goal of the IT audit is to ensure that the IT infrastructure, supporting controls, policies and procedures provide security, maintain data integrity and adequately support the organizations goals and objectives. Ultimately, the audit process provides confidence to stakeholders, including shareholders, banks, and regulatory authorities.

IT audit processes will be familiar to anyone with experience in financial auditing and include planning, a review of controls, evidence collection, and reporting. The IT audit planning phase includes identification of all the elements to be evaluated and the potential material impact of these elements on the organization’s ability to achieve its goals. Control evaluation includes a review of systems security policies, reconciliation processes and methods to ensure data integrity. Evidence collection might include sampling of system data, actual observation of transactions, or systems monitoring.

While the principles behind the IT audit are sound, the execution is often poor. To support the demands of the IT audit process, audit firms employ staff specifically knowledgeable and certified in information systems audit control best practices. Unfortunately, IT auditors are usually schooled only in general audit practices and principles, and not in particular application technologies or platforms. In short, IT auditors understand best practices, but have little practical knowledge or experience in dealing with the specific systems, or even similar systems, to those being evaluated.

It has been my experience both an as IT systems auditor (I am CISA certified and worked as an IT auditor for one of the Big 4 firms for several years) and as an Operations Manager, that IT audit staff are rarely provided training on industry specific toolsets. These toolsets are often integral to an organization’s ability to conduct business, and thus are of greater materiality, and must be closely scrutinized. The IT audit framework does outline Computer Assisted Auditing Techniques for substantive testing of application and database controls. These tools, however, are also generic, and do not test underlying business assumptions – perhaps the most important component of an application. These tools are useful, but should not be relied upon to test controls of custom applications which are completely unfamiliar to the auditor.

During our last year’s annual audit, the IT auditor, who is CISA certified, asked that I provide test cases for him to perform control tests on our internal billing system because he did not understand the fundamental principles behind our billing system and had no previous experience with systems and processes specific to our industry. The audit firm provided equally as inexperienced staff for the previous year’s IT audit. This is obviously unacceptable — to me and to our shareholders, and does not meet the quality standards set by audit best practices.

An IT audit should provide benefit not only to stakeholders, but to the organization as well. Correctly performed, the audit provides an opportunity to identify and correct inefficiencies, improve product and service quality, and better service customers. To this end, it is essential that management and senior technical staff be actively involved in all aspects of the audit.

Consider the following:

  1. Insist on IT auditors with experience in your industry. Telecommunications billing systems, for example, require an understanding of call flow, rate planning and management and basic networking. An IT auditor, attempting to test whether a call rating engine is accurately charging, would need a fundamental understanding of Call Detail Records. Without this knowledge, the auditor would have no insight into why calls are rejected by the switch and others, for example, by the call rating engine.

  2. Document all aspects of the IT audit. This means keep records of all data provided to the auditors, of all communications with the audit team and all systems documentation provided. IT systems generally do not change on an annual basis, but your audit team will! By keeping copies of information provided for previous audits, internal management can expedite the new audit team education process. Keep in mind as well that you are paying for the new audit team to learn about your systems. This type of documentation can help reduce audit costs. Types of documentation might include: network diagrams, application and process flow models, database schema, systems documentation, change logs, reports, records of system errors and evidence of regular maintenance.

  3. Actively participate in the planning process. Operations Management should review the list of all systems, process, interconnects, procedures, and policies to be audited. Elements that have changed since previous audits need to be noted and explained to the audit team. As part of the planning and risk assessment process, the auditors should present the controls that will be tested and what methods will be used to test. This plan must directly pertain to the organization’s IT infrastructure and supporting environment.

  4. Dedicate knowledgeable internal IT resources for assisting with the audit. These resources should be familiar with all aspects of the IT infrastructure and environment. Their purpose is to expedite the audit process and ensure that all items covered on the IT audit plan are executed and tested as planned.

  5. Assess the audit. Provide a copy of the IT audit report to operations and functional managers. This enables management to fully understand and redress issues outlined in the report, as well as review accuracy and ensure completeness. Discuss specific technical issues and assumptions with the staff responsible for managing the system or process.

Finally, at the close of the process, update the IT audit documentation file. Review internal issues associated with the audit. For example, was the audit staff knowledgeable in your industry? What specific systems or controls should have been more closely evaluated and why? Did staff spend too much time focusing on a particular system or process? What was the result of this effort? By being proactive, the IT audit process can be executed quickly and provide substantial value to the internal departments responsible for the organizations IT.

For additional information on IT Governance, visit the Information Systems Audit Control Association.

Data warehouse risk management – risk identification

Data warehouse projects are highly complex, and as such, are inherently risky. It is the responsibility of the project manager to lead the data warehouse team in identifying all risks associated with a particular data warehouse implementation. The goal of this process is to document all essential information relating to project risk.

For additional information relating to risk, refer to the Project Diva article on project risk identification and management.

 

Risk ID

 

Category

 

Description

 

Conditions

 

Impact Description

 

Possible mitigation

 

1

 

Overall

 

Loss of project sponsor

 

Impending reorganization

 

Scope, budget, staffing, schedule issues

 

Identify secondary sponsor. Review project requirements with
sponsors

 

2

 

Overall

 

General lack of experience with toolsets, methods and best
practices

 

Potential for huge margins of error in budgeting and scheduling;
project delays; deliverables not fit for use

 

Strong resource management plan; clearly defined project
responsibilities; recruiting staff with DW experience; off-site training;
professional consulting

 

3

 

Scope

 

Wide range of users driving system requirements

 

Conflicting user requirements

 

Scope creep; application not meet user requirements; marginalized
users

 

Ensure high level of user involvement; requirement
prioritization

 

4

 

Scope

 

Changing system requirements

 

Impending reorganization; new product development; staff
turnover

 

Schedule delays; system not meet business
objectives

 

Change management process; project sponsor
involvement

 

5

 

Budget

 

Inadequate Budget

 

Project delay; scope scaled back; not meet business
requirements

 

Research; professional cost estimation; contingency budgeting; sponsor
support

 

6

 

Schedule

 

Unrealistic schedule due to initial estimates / poorly understood
deliverables

 

Large initial delays; evidence of tasks excluded from
WBS

 

Project delay; quality issues due to rushed delivery or exclusion of
deliverables

 

Research; sponsor support; professional
consulting

 

7

 

ROI

 

Disconnect between business objectives and project
deliverables

 

Changing business objectives; change in management; market
changes

 

Product not fit for use; no ROI

 

Proactive alignment of project with business objectives; active
stakeholder management; prototyping

 

8

 

Technical

 

Scalability issues due to huge amounts of data, changing
requirements

 

Poor system performance

 

Data access issues

 

Estimating toolsets; technical design completed by experienced
DBA

 

9

 

Technical

 

Support issues; heterogeneous environment; new
technologies

 

Staff turnover; staff currently not trained on
systems

 

System unavailable

 

Administrator training; staff training budget; DR
plan

 

10

 

Technical

 

Poor data quality

 

Useless datasets; not meet business requirements; system not
used

 

Data quality review sub project; data cleansing; implement known clean
data first; metadata quality tags (e.g. “questionable,” “not yet
reviewed”)

 

11

 

Technical

 

End user technical skills too low

 

Implement canned reports and templates; user involvement in front end
design

 

12

 

Implementation

 

Vendor issues

 

Budget; project delays

 

Verify vendor financial viability; vendor
references

 

13

 

Implementation

 

User rejection

 

System not used; users express dissatisfaction during training,
deployment

 

High level of user involvement; prototyping; user feedback incorporated
into project; user training

 

Data Warehouse Risk Identification (.doc format)

Call Center Setup Project Planning

Call Center Setup

Call center models range from the simple internal IT help desk, to a fully staffed customer care department, to the more complex multi-channel systems with IVR (Intelligent Voice Recognition) and ACD (Automatic Call distribution) call handling.

All call centers will require software to track and manage customer issues, a voice and data network to support the software, and adequate staff to handle call volume.

The following project documents cover varying aspects of call center setup and provide a project management framework for successful deployment of a call center.

Project Documents

Call Center Software Setup Example WBS

Call Center Software Requirements Definition

Call Center Setup Project Risk Matrix

IT Disaster Recovery Project Objectives

  • Document all existing IT practices, procedures, standards and guidelines by June 2016.

  • Inventory and document all IT systems, software and supporting network infrastructure by July 2016.

  • Identify and document all 3rd party IT service and support partners; provide contact information and copies of associated contracts by July 2016

  • Establish an IT Disaster Recovery Committee to review accepted practices, procedures, standards and guidelines defined as a result of this project and prioritize importance of IT systems by August 2016.

  • Establish a formal and documented Disaster Recovery Program by September 2016.

  • Establish a redundant data center, and deploy those services identified as high priority by the Disaster Recovery Committee by October 2016.

  • Deploy those services identified as medium priority to the redundant data center by December 2016.

  • Implement Maintenance and Operations plans that ensure on-going, annual review of the IT Disaster Recovery Programs established by this project by December of 2016.

  • Develop and distribute IT Disaster Recovery Plan documentation to management and staff members identified by the Disaster Recovery Committee by December of 2016.

Website Development Contract Considerations

Aside from the standard clauses (liability, force majeure, cancellation, etc.), a website development contract should specifically address the following:

Scope

Many website development contracts will be signed before the project scope has been fully defined. In cases such as this, the contract should require the development firm to fully document project deliverables, budget and schedule before formally beginning work on web site. Contractually, this could be considered the first project milestone. If this milestone is not reached (and the contract would need to specify criteria for acceptance of this and all project milestones), then the contract would be void.

If a scope of work has been developed, then, as part of the project plan, the project deliverables need to be included in the website development contract. The scope of work includes all work to be done as part of the project. Items not specifically included in the scope of work are considered out of scope and would be addressed as part of a change management process. A website development project scope of work might include:

  • Design – site graphics, layout, color schemes, logos, images, illustrations, maps, etc. If these are to be developed by an external design studio, then issues of copyright and use will specifically need to be addressed as part of a development contract. If the art works are to be provided by the customer, then the contract should specifically state this. The contract should specify the designs to be provided in as much detail as is possible. For example, “the development house will design 3 proof logos based on existing marketing materials and corporate color schemes for review by the client.”

  • Content – content includes all text, media, downloads. Content development is perhaps the most time consuming task in developing a web site. Content is extremely important! Sites that plan on deriving revenue from traffic will need content to attract and retain site visitors.

  • Supporting applications – most websites will utilize applications to support content delivery and information collection. These may include bulletin board systems, forms, database systems, shopping carts, product catalog management systems, etc. Many of these applications are available as open source. The website contract should specify the source of the applications, whether open source or custom development and document any license restrictions or rights associated with these applications. In instances of custom development, license and right to use should be specified in the contract.

Change Management

Change management procedures outline how changes from the initial scope are handled by both the service provider and the customer. Most change management policies call for documenting the nature of the change and allowing time for proper evaluation of the costs of change and subsequent impacts on schedule, quality and risk. In the case of a website development contract, particular attention should be paid to managing design changes, as these can be both time consuming and expensive.

Project Responsibilities

A responsibilities clause provides clarity to the various roles and responsibilities of team members and management involved in the website development project. If the client is responsible for generating content, for example, then this should be documented under the responsibilities clause. The contract should cover at the very minimum: design, media and content sources, search engine optimization, testing, maintenance, hosting.

Payment Terms

Most contracts that cover a technology deliverable will use milestones for determining the payment schedule. It is essential that these milestones be clearly defined, with documented milestone acceptance criteria. An example milestone for a website development contract might be delivery of design proofs, completion of content, or successful testing of a data management application.

User Acceptance Testing

Testing is essential, as a company’s website is often the first interaction that a potential customer, partner or employee has with the organization. Like all company documents, sales and marketing materials, the website must be free from error. Completion of testing should signify a project milestone.

Warranty

A warranty period will ensure that underlying web applications, components, scripts and other site developments are free from error for a period following project sign off by the client. The contract should detail the specifics of the warranty (e.g. length of warranty, what elements are covered by the warranty, time frame for remedy and what costs, if any, will be incurred by the client in the event that warranty repairs are required).

Ongoing Responsibilities

If the development house is responsible for site maintenance, then a regular maintenance schedule should be included in the contract, as well as a listing of the items which are covered by the maintenance agreement (e.g. data archiving, product catalog updates, content updates, design refreshes).

Copyright

In addition to underlying web applications, scripts and source code, the contract will need to cover copyright of site design, logos, graphics, illustrations, images or other media produced for the website.

Project Timelines

The development contract should specify in as much detail as possible the proposed delivery schedule. If a formal scope of work has not yet been established, then defining a project timeline will not be possible. Rather, the development contract should be used to specify expectations and state that as part of the formal project scope, a project schedule will be created for the client to review.

The most important part of the website development contract and procurement process is having a legal expert who is familiar you’re your business and the project review the contract!

Project Risk Register

The Project Risk Register is a summary risk document. The risk register should contain a listing of all sources of risk, risk impact, risk probability, contingency plans, risk owner and the status of the risk. The risk identification, impact assessment and risk ranking, contingency planning and project execution tasks are all inputs to this document.

The risk register should be actively maintained throughout the project. The below example project risk register is based on a generic website development project.

Risk ID

 

Description

 

Risk Impact

 

Impact Description

 

Risk Probability

 

Risk Severity

 

Risk Contingency

 

Risk Owner

 

Status

 

Date

 

Content will not be available when required 

4

 

Missed deadline will result in funding cut. 

3

 

12

 

Hire additional copy editors 

Marketing 

Currently recruiting for additional staff 

10/10/08

 

Project resources may be reallocated for other (higher) priority
project 

3

 

Resources are currently 100% committed to this project. Losing a
resource would severely delay project. 

2

 

6

 

Work with Portfolio Manager to ensure resource availability 

PM 

Portfolio Manager will not reallocate staff. 

10/11/08

 

End users may be resistant to new site workflow 

4

 

Customer feedback was instrumental in site design, but not all
customers will like the site changes. May result in bad publicity or
customer satisfaction issues. 

3

 

12

 

Implement online help facility. Provide additional customer support
staff for transition. 

Support Manager 

Additional staff will be brought online 1 month prior to release.
Online help facility added as change request. 

9/9/08

 

Data loss 

5

 

Data migration may result in loss of customer data. Customer data is
key for day to day business operations. 

2

 

10

 

Revisit backup procedures. Execute manual backup prior to
migration. 

Developers 

   

Call Center Software Project Requirements Definition


1. General Call Center Software Project Requirements

1.1. Scalable to support multiple call center locations
1.2. Track both inbound and outbound customer communications

1.2.1. Provide performance metrics and reporting on call queue
1.2.2. Provide performance metrics on staff call handling
1.2.3. Associate all communications with appropriate customer account(s)

1.2.3.1. Maintain log and provide management and center staff access to historic call data

1.3. Call Center software must integrate with CRM (Customer Relationship Management) application

1.3.1. Support screen pop based on dialing number for existing customers

1.4. Call Center software must integrate with Billing System
1.5. Call Center software must integrate with Help Desk application to

1.5.1. Track and manage open issues
1.5.2. Escalate and/or dispatch issues to appropriate department (e.g. level II support, engineering, management)
1.5.3. Support the work order management process
1.5.4. Provide incident statistics and management reporting

1.6. Call Center software must provide Knowledge Database

1.6.1. Automatically update knowledge database with issue and resolution information
1.6.2. Provide administrator facilities for managing knowledge database
1.6.3. Knowledge database should be searchable and available to customers via the WWW

1.7. Call Center software Quality management

1.7.1. Support the ability monitor calls for call quality
1.7.2. Support the ability to record calls

1.8. Comply with e-911 standards and provide support for lawful intercept

2. Call Center Software Email Routing Requirements


2.1. Support intelligent routing based on customizable rules definition
2.2. Ability to associate email with customer account


3. Call Center Software Voice Routing Requirements

3.1. Use ACD (Automated Call Distribution) and IVR (Intelligent Voice Routing) to direct calls
3.2. Use Auto Attendant features to greet customer based on time of call

3.2.1. During business hours
3.2.2. After business hours
3.2.3. Holiday
3.2.4. Outage notification
3.2.5. Special announcements

3.3. Routing, Auto Attendant, new extension setup should be manageable by in house technical staff
3.4. Support WWW to phone call center applications

Example Project Assumptions for a Data Migration Project

Example Project Assumptions for a Data Migration Project.

Data Migration projects, especially those across platforms or into new database structures, are inherently complex. Identifying project assumptions early in the project planning process is necessary to facilitate the scope of work definition. Assumptions should be documented clearly, without use of department specific acronyms or terminology which might not be understood by all members of the project team. Project assumptions, especially those initially identified by the Project Manager without input from the rest of the project team, should be distributed to the whole team as early in the project planning process as is reasonable.


Example Billing Project data migration project assumptions

  • Business rules relating to billing processes (e.g. invoice cycles, pro-rating logic, re-connection and other fee structures) remain the same. A summary of used business rules will be distributed by the Finance Department.
  • Data for active billing subscribers will be migrated. As at (date), there are 1000 active subscribers.
  • Data for recently disconnected subscribers will be migrated. As at (date), there remain 250 inactive subscribers. Given current churn rates, there is a high probability that 75% of these currently inactive subscribers will request re-connection.
  • Data for customers with an account balance (debt or credit) will be migrated.
  • Data for customers with un-returned equipment will be migrated
  • Data for customers currently in the collections queue will be migrated.
  • Data to be migrated includes (refer to data dictionary for format and detailed description of data to be migrated):
    • Customer name
    • Customer contact details
    • Customer billing package details
    • Customer equipment details
    • Customer invoices generated since (date)
    • Current, 30, 60, 90, > 90 day balances
    • Notes on customer account
    • Service call history details
    • Method of Payment details (Credit Card, Direct Deposit details)
  • Data to be migrated must be exported and cleaned before formatted for import into new platform. Cleaning can be completed by the Customer Service team before the end of (date).
  • Data transformation can be completed by the Billing Administration department.
  • The transformation tools identified and tested by the Billing Administration department will be adequate for the scope and duration of the project.
  • Existing and new platform documentation is accurate.
  • Internal staffing resources will be made available to complete required system testing.

Data Center Relocation Work Breakdown Structure

The purpose of this document is to provide project managers and team members with an overview of the requirements for a data center relocation. This data relocation plan document can also be used as a checklist in support of an existing data center relocation project.


1.1. Data Center Relocation Project Management

1.1.1. Scheduling

1.1.2. Procurement Management

1.2. New Data Center Requirements Definition

1.2.1. Business


1.2.1.1. Service Level Agreement

1.2.1.2. Insurance

1.2.2. Technical


1.2.2.1. Current Data Center

1.2.2.1.1. Inventory

1.2.2.1.2. Communications Migration Requirements

1.2.2.1.3. Contract Review

1.2.2.2. New Data Center

1.2.2.2.1. Rack Diagrams

1.2.2.2.2. Communications Interconnects

1.2.2.2.3. Cabling

1.2.2.2.4. Power

1.2.2.2.5. Environmental Controls

1.2.2.2.6. Security Access

1.3. Acquisition

1.3.1. New Data Center


1.3.1.1. Site Visit

1.3.1.2. Contract

1.3.2. Hardware


1.3.2.1. Issue RFP

1.3.2.2. Selection Criteria

1.3.2.3. Purchase

1.3.3. Communications


1.3.3.1. Data Interconnect

1.3.3.2. Voice Interconnect

1.4. Installation Prerequisites

1.4.1. Resource Allocation

1.4.2. Notifications


1.4.2.1. Internal

1.4.2.2. Customer

1.4.3. Site Preparation


1.4.3.1. Circuit Cut Over Plan

1.4.3.2. Rack Space Allocation

1.4.3.3. Power

1.4.3.4. Cabling

1.4.3.5. Network

1.4.3.5.1. IP Allocation

1.4.4. Testing

1.4.5. Configuration Backups

1.4.6. Detailed Installation Documentation

1.5. Migration

1.5.1. Hardware


1.5.1.1. Installation

1.5.1.2. Testing

1.5.2. Routing and Switching


1.5.2.1. Cut Over

1.5.2.2. Testing

1.5.2.3. Cancel Circuit to Old Data Center

1.6. Operations Switch Over

1.6.1. SLA monitoring

1.6.2. Documentation

1.6.3. Disaster Recovery Testing

Microsoft Office Format of Data Center Relocation Plan Work Breakdown Structure

Office Relocation Example Risk Matrix

This example risk matrix uses the Office Relocation project assumptions and Office Relocation work breakdown structure documents as input. This is not an
exhaustive list of all Office Relocation project risks and may not be applicable to your particular project. Rather, this matrix is intended to provide a framework for
identifying, prioritizing and managing risk.

Risk Description

Impact

Probability

Severity

Mitigation Strategy

Priority Scale 1-5 (5 = most severe, greater probability)

(A)

(B)

(A*B)


Technology:  Data and/or Voice circuits will not be available by (date).  Company will not be able to communicate. (Note A)

5

3

15

1. Contact wireless solution vendor as redundant provider.
2. Seek alternate fixed data / voice provider
3. Allow employees to work from home
4. Extend existing lease by 1 mo nth to provide additional circuit implementation time.

Operations:  Inadequate insurance.  Company may be underinsured.

4

1

4

1. Check coverage
2. Review policy

Construction:  Construction build not complete by (date) (Note B).  Company will not be able to relocate according to schedule.

5

3

15

1. Delay lease termination notice

Construction:  Construction deficiencies.  Office environment may be hazardous.

3

2

6

1. Construction warranty
2. Phased payments
3. Use known good contractor
4. Request inspection

Move:  Equipment breakage.  Core networking gear and/or peripherals would need to be replaced.

5

2

10

1. Adequate packing
2. Hire professional packing/moving company
3. Employees responsible for laptops
4. Ensure spare equipment available

Note A: Historically, vendor has had difficulty deploying high speed circuits to this part of town.
Note B: Project is currently 1 month ahead of schedule. However, project is complex, so some delays are expected.

For additional reference, review the Project Risk Register and Risk Impact and Probability Assessment documents.