Executive Summary
This report summarises the evaluation of a Pilot project focused on using the BRANZ Artisan tool for the creation of digital records of low-risk plumbing and drainlaying (plumbing) work.
The evaluation included a review of 93 digital records of plumbing work by an independent subject matter expert, and 13 interviews with BCA inspectors and plumbers participating in the pilot. The key limitations of this evaluation are that the Pilot was tested in only two BCAs, and participation was only offered to high-performing plumbers. As a result, care should be taken when using these findings to make statements about plumbers nationally.
Artisan is an effective tool for creating digital records of low-risk plumbing work
BCA representatives and plumbers interviewed for this evaluation were unanimously positive about their experience of using Artisan to create digital records. Following a review of a sample of digital records, an independent subject matter expert also felt that Artisan enabled production of high-quality digital records.
BCA interviewees noted that the use of Artisan for creating digital records, and reviewing them as part of remote inspections, was a good fit with the largely digital processes involved in the building consent system. BCAs interviewees felt that there could be pushback from a small number of inspectors who prefer to work face-to-face on building sites and could struggle with inspections being undertaken remotely as a desk exercise.
All plumbers interviewed felt that Artisan was simple to learn and that the process of creating digital records was intuitive. The creation of digital records is an extension of existing practices for many plumbers, who already take photos of their work. The benefits Artisan offer over current practice is that the photos are organised, able to be annotated, easier to reference in the future if required, and that the digital records are recognised by the BCA enabling remote inspections of work.
A key factor in the success of Artisan, and uptake by plumbers, is in the level of buy-in from BCA staff.
While the digital records reviewed were high quality, most of them had aspects which could be improved upon – notably missing information
The independent reviewer noted that while they felt the digital records created in Artisan were of a high quality, most of the records (around 80%) could be improved. Most of the issues identified with the digital records related to missing information: 32 of 93 records were assessed as ‘not covering the full scope of work’, and 67 of 93 records were assessed as ‘not having sufficient notes or comments’. The missing information in question was typically either minor or likely to be included in other building consent documentation.
It's important to note that the independent reviewer was assessing the digital records in isolation. In contrast when making determinations about compliance with the building code, BCA inspectors will have a range of other information sources (e.g. contextual information about the work, supporting documentation, and conversations with plumbers and other stakeholders).
One BCA interviewee described how the process of using digital records to determine compliance largely mimics the process of in-person inspections. Inspectors won’t conduct detailed inspection and testing of every component but will instead look at a sample of components and make a determination of compliance. Artisan includes a ‘shot list’ for each job, directing plumbers to take a series of specific photographs, providing a sample of work for the BCA inspectors to review.
Creating and using digital records provides significant efficiency gains for plumbers and BCA inspectors
BCA respondents and most plumbers interviewed noted that using Artisan to create digital records results in significant efficiency gains.
Digital records created in Artisan significantly reduced the time BCA inspectors spent traveling to and between building sites to conduct inspections. One BCA respondent also noted that when conducting an in-person inspection, a failure would almost always require scheduling and undertaking another inspection to inspect the failed elements. Whereas a failure in Artisan could be resolved more quickly, for example through requesting a better angle on an existing photograph. As a result, a remote inspection using a digital record created in Artisan could be replacing two or more in-person inspections and the associated travel time.
A widely shared view among the plumbers interviewed was that Artisan was more time efficient than the traditional method of in-person site inspections. Respondents noted that using Artisan to create and submit digital records was significantly faster than booking, and waiting for, in-person inspections. While the detailed shot lists required some effort to compile, feedback from plumbers interviewed was that the benefits of using Artisan far outweighed the small time cost associated with ensuring the digital record was complete.
All stakeholders described how the use of Artisan to create digital records had supported increased flexibility in how they structure their workdays.
Some stakeholders felt the high-trust nature of digital records and remote inspections introduced risk associated with misuse of the system
Several plumbers noted the potential for misuse of the Artisan system. A key concern was that it was possible to meet the requirements of the shot list set by the BCA while still misrepresenting or omitting relevant details (e.g. the fall of a pipe, or adjacent pipes that may pose a risk). To mitigate this concern, some respondents suggested quality assurance measures such as random in-person inspections.
BCAs were less concerned about this, primarily because they saw access to Artisan as something they controlled rather than being open to all. BCAs selected pilot participants based on their demonstrated capability, for example low fail rate over a sustained period. One BCA noted that they see Artisan as a useful tool to streamline how they work with their high-performing plumbers, and not as something that should be rolled out to everyone. They commented that if bad behaviour was detected, for example people looking to cut corners and reusing photos, they would have their ability to use Artisan removed and return to in-person inspections.
This finding raises the issue that if Artisan is provided as an option to all plumbers, rather than those selected by BCAs, sufficient checks and a functioning sanction system should be in place to ensure no misuse of the system.