Important changes coming from the PCAOB

Jeff Hare has been working in the field of Audit and Compliance for over 25 years, and has extensive experience with ERP Applications and their controls. Jeff has written several books on securing your applications and from recent training provided to the PCAOB put together the following article.

ERP software provides organizations with tremendous benefits including vast configurable processes and standard reports used for reporting on data. ERP software comes in two flavors: those that organizations need to manage their own infrastructure (on-premise software providers) and those where the infrastructure is managed by the software vendor (SaaS software providers).

Software providers (SaaS and On Premises) are subject to audits where an independent auditor reviews and tests the controls defined by management and issues an opinion on their control environment. These audits are referred to as SOC (Service Organization Controls) audits. Part of the SOC audit scope is testing of a population of reports provided by the software provider (what we refer to as “seeded” reports). Management routinely uses seeded reports in the execution of their controls.

Management is responsible for documenting how they concluded that the reports they are using are complete (C) and accurate (A). Auditors refer to this as C&A testing. This is true for seeded (as referred to as ‘default’ ‘standard’, ‘canned’, or ‘vendor-supplied’) reports as well as any custom reports or queries they develop. External auditors are required by the PCAOB to evaluate and test management’s controls over such reports.

For seeded reports, the review by external auditors has historically been limited to obtaining the SOC report. The SOC report provides general certainty over the software provider’s change control process. As reports are developed or changes are made to the reports, they must be subject to the software provider’s change management process.

The change management process should include controls such as:

• The developer of the code should be separate from the person (promoter) migrating the code to production.

• The promoter of the code needs to move the code into a secure environment that the developer does not have access to – what we refer to as a User Acceptance Testing environment (UAT).

• A test script is prepared and executed by a person or group (tester) doing the testing.

• Testing should verify the completeness and accuracy (C&A) of the report.

• Documentation of the successful test should be kept as evidence associated with the change ticket in the software providers change management ticketing system.

As part of the SOC audit, the audit firm would pick a sample of actual changes and ask the management of the software company to provide evidence that the changed report was properly subject to the change management process.

External Audits

When the SOC auditor issues their opinion on the software company’s controls, it is based on the sampling it took during the audit. The SOC auditor does not test every change, only a sample of the changes. Historically, the SOC auditor does NOT provide the details in their SOC report of what reports were sampled.

When external auditors perform their audit, they look to management to identify a population of key reports that are being used to execute manual controls. Management, ideally, should be identifying this population by mapping it from the key controls in their Risk and Controls Matrix (commonly referred to by auditors as the RCM or RACM – pronounced “Rackum”).

External auditors should be testing the Completeness and Accuracy testing of these ‘in scope’ reports. Historically, any report that is provided by the software vendor, referred to as “seeded” reports, gets a pass by the auditor because it is assumed that the vendor tests new reports and changes to existing reports as part of their change management process. Even in these circumstances we believe it is prudent for management to test and document their own C&A testing because the software company does not disclose its methodology and management still needs to confirm the data is appropriate for their specific control.

Even though the SOC auditor does not test ALL reports, they test a sample of the reports. If the sample they pick has adequate change management controls, the software company’s change management system is deemed to be effective over all reports.

Transitioning now to what we are hearing from external auditors is a change in the expectations from the PCAOB. The risk being presented is that the sample the SOC auditor took did not include the specific reports that management is using. The expectation now is that management confirm with the software company or the SOC auditor whether the report(s) they are using to execute controls were specifically tested. This is a significant change in expectations and could cause a significant increase in scope for the SOC audit.

Changes to reports.

There is another risk that needs to be considered as well; the risk that the report is changed by the software provider. Most current SOC reports do not specify the reports that were samples and they also do not provide a list of reports that have been changed when they patch their applications. SaaS software providers typically patch their applications either semi-annually (2 times per year) or quarterly (4 times per year). Software providers of on-premise software provide patches, but not necessarily on a regular basis. Organizations often have the choice to defer applying patches to the software application for years such that the risk of changes to seeded reports is much less frequent than SaaS software providers whose patches are not optional and cannot be delayed.

Management should be re-performing their C&A testing whenever a report definition is changed. Therefore, it would be ideal for the software company to disclose the reports that have had a modification to the logic when they release each patch (often referred to as a ‘read me’ file). This would provide management a notice that they need to re-perform their C&A testing.

Following is a summary of the current risks that are often addressed by management and the suggested controls management needs to consider implementing:

Following are our recommendations overall:

1. Management needs to identify a population of key reports from the Risk and Controls Matrix. This includes controls that support manual controls (referred to as IT dependent controls) and IT general controls. This population MUST include all reports (seeded or custom) and queries that are used to execute any control.

2. Given software providers do not identify the specific reports, management should baseline all reports – seeded or custom. Whenever a report is changed in any way, management should re-baseline their C&A testing.

3. For seeded reports, given software providers typically do not disclose when they have modified seeded reports as part of their release notes, we recommend that management re-baseline and document the C&A testing for each report each time the software provider patches the applications.

4. For custom reports and queries, management needs to put in place the following controls:
* All changes to reports and queries need to be subject to the change management control. This includes re-baselining the C&A testing prior to changes being made in production.

5. Management needs to put in place a pro-active monitoring process to identify when any changes are made to custom reports or queries. This would include logging changes to the OS files, programs and files stored in the database, report definitions, changes to input controls such as a list of values, and any other place where logic / code in a report may be changed. This set of logs would constitute a change management population for changes to key reports. Management should regularly review, no less than monthly, this change management population to identify if any key reports were changed. If so, they should confirm these changes were subject to the change management process and that the C&A testing was re-baselined.

Summary and Conclusions

The PCAOB has asked some reasonable questions about ‘in scope’ reports being provided by software providers. Management needs to consider updates to their ITGCs to accommodate this change in expectations. Management should also request from their software providers to be more detailed in their SOC reports including specifying which reports are being tested and to include testing of specific reports as part of their SOC audit.

Effective Controls

If you are utilizing the delivered Security and Configuration within your Applications, there is a good chance you have Segregation of Duty violations. Seecuring will assist you with establishing and reporting on the issues you have through to helping remediate the issues (which usually represents the biggest and most time consuming aspect of implementing internal controls). Specifically, Seecuring delivers:

• Segregation of Duties

• Sensitive Access

• User Access Reviews

• Patch Impact Analysis & Configuration Changes

We have been working with ERP/HCM Applications since the early 2000's, and work with leading CPA's, Audit staff and Application specialists to deliver a complete solution.

Before you invest in expensive Software, why not look at GRC as a Service? Faster delivery, lower cost, and more than just reports on your issues - we help Organizations achieve their goals for Internal Controls.

To discuss your requirements, you can schedule a call with us: