Approach
The evaluation utilized two key methodologies in collecting data:
- Third-party quantitative analysis of a sample of digital records captured using the Artisan tool, undertaken by a subject matter expert.
- Stakeholder interviews with relevant BCA staff and plumbers.
Analysis of a sample of digital records
The analysis of a sample of digital records was intended to provide a quantitative account of the extent to which digital records enabled determinations of compliance to be made.
BCAs provided access to digital records of a sample of plumbing work entered into the Artisan system. These records were reviewed in detail by a third-party subject matter expert, with significant experience as a plumber, as well as in multiple roles associated with the building consent system. Rather than being a random sample, cases provided to the third-party expert were purposively sampled to include a range of job types, as well as a mix of cases that passed on their first submission and those that required additional clarification from plumbers before a determination of compliance could be made.
A total of 93 cases were reviewed by the third-party expert. The information included in each digital record was assessed using five questions exploring the quality of the digital record:
- Is the visual clarity of all the photos sufficient to assess the work?
- Do the photos cover the full scope of what is being assessed?
- Is there sufficient information to identify any environmental, structural or safety risks associated with the work?
- Is there sufficient information to understand what each photo refers to?
- Are there sufficient notes/comments where needed to support demonstrating compliance?
These five questions were then summed up in an overall assessment:
- Is there room to improve the quality of the digital record to demonstrate the work complies with the building code?
It’s important to note that this assessment was not assessing whether it was possible or appropriate to have made a determination of compliance. Each case had already been assessed by a BCA, and a determination of compliance made. This assessment is focused on the quality of the digital record as a stand-alone article.
Stakeholder interviews
The stakeholder interviews were intended to draw out narratives from the two key stakeholder groups: Building Consent Officials and plumbers. These narratives provide evidence about the extent to which Pilot objectives are being achieved.
Interviews were conducted with a senior Building Consent Official most suited to discuss the use of the Artisan tool from each of the two participating BCAs. BCAs were asked to provide a list of plumbers who were using the Artisan tool, from which to select interview respondents. A total of 11 plumbers were interviewed across the two regions included in the Pilot. Interviewees were selected to ensure a range of experience levels, from trainee plumbers through to experienced certifying plumbers.
Topics covered in the plumber interviews included:
- their experience using Artisan to create and reference digital records
- benefits and costs of using Artisan – efficiency, how they structure their day, interaction with the council
- any changes they would make to Artisan or how it is used for creating digital records.