Data Engineer
On this page
Tēnei tūranga – About the role
The Data Engineer is a technical position in the Joint Targeting and Analytics team within the Immigration Risk and Border (IRB) branch of Immigration New Zealand (INZ). The IRB branch works to collaboratively identify and mitigate risk early and to reduce harm to the integrity of the immigration system, system users, and New Zealand.
The Joint Targeting and Analytics (JTA) team offers specialist expertise in the analysis of data and information to identify and target immigration risk. Services provided by the team include the delivery and management of the Visa Triage process, risk identification and targeting at the NZ border, and the provision of risk models and analytics.
The Data Engineer supports the JTA team by managing and optimizing the risk analytics environment (Databricks or Snowflake), managing the JTA raw data environment, and supporting JTA data scientists and data analysts in delivering analytical solutions to business problems and queries This may include ingestion of new data sources, building and optimizing ETL processes, researching and answering questions about how to access and interpret data, and providing coaching to JTA and the wider network about how to write efficient and sustainable SQL queries.
The role actively engages with the data warehouse team and keeps up to date with the latest developments within data engineering to ensure INZ uses the most appropriate tools and methods and incorporates these to existing processes within INZ.
Ngā herenga – Requirements of the role
Personal specifications
- Demonstrable experience of data warehousing, business intelligence and analytics solutions in a complex data environment.
- A demonstrable competency in the use of SQL programming and Python.
- Strong development experience within cloud environments, preferably Azure/Snowflake.
- Strong experience with developing and deploying efficient workflows at enterprise scale.
- A track record of understanding how real-world business processes are manifested in enterprise data.
- A strong view on SQL development best practice, data modelling and database design.
- Strong commitment to documenting analytical projects and outputs.
- Excellent communication skills and the ability to develop solid working relationships.
- Sound knowledge of architectural and design techniques, industry standards and practices.
- Good understanding of data governance principles in a public sector environment and the ability to apply and operate within the relevant legislative framework.
- Very process oriented and detail minded and able to work with and understand security, privacy and ethical requirements.
- Excellent written and verbal communication skills.
- A relevant tertiary qualification, preferably in software engineering or a similar discipline.
- Can gain and maintain a national security clearance as required.
- Must be a New Zealand citizen or a permanent resident.
Takohanga tuhinga o mua – Key accountabilities and deliverables
Delivery Management
Maintain and optimize the JTA risk analytics environment including:
- Manage user access to databases to ensure security and timeliness requirements are met and projects can progress efficiently
- Monitor and maintain ETLs to ensure various existing and new data sources are combined and stored correctly and reliably for use in existing and new data science products
- Monitor and maintain back-ups to ensure business continuity
- Monitor logs and provide reporting and recommended actions when necessary
- Ensure data consistency, accuracy and cleanliness to maximise the effectiveness and efficiency of team operations and outputs
Manage the ingestion of new data into the risk analytics environment
- Design data models to hold new data sources
- Build sustainable ETL processes to ingest new data sources and keep them updated
- Clearly document new ETL processes
- Join JTA project teams to support new initiatives.
Engage with MBIE Central Data team to support continuous improvement in data management and governance
- Support the development of extraction processes to create required datasets from data held in JTA data environment.
- Produce appropriate documentation to enable effective future support.
- Inform requirements for changes to the JTA analytics environment to hold new data required for analytical activities.
Enable the JTA team
- Provide advice on the potential impact of any proposed solution designs on the ability to deliver integrated data sets to JTA.
- Undertake trouble shooting and bug-fixing for developed solutions during testing and subsequent activities. This includes providing operational support to help ensure that JTA deployed systems meet required service levels.
- Act as a primary point of contact between JTA and the MBIE Data Warehouse team with regard to the provisioning and preparation of an integrated analytical data set.
- Act as the subject matter expert (SME) on the processes, technology and protocols for the preparation and integration of analytical data sets.
Examples of tasks assigned to the Data Engineer include:
- Curate existing data, creating refined ‘silver’ and ’gold’ tables
- Lead migration and replication of data from source systems/MS SQL Server into the JTA analytics environment
- Aid in entity resolution initiatives harmonising data from different sources
- Lead implementation of graph databases along with associated ETL workflows
- Work with data scientists to develop pipelines generating clean, consistent, and re-useable features stores ready for machine learning algorithms
- Establish operational data quality checks to catch, report, and sanitise malformed data before it enters reporting ecosystems.
Wellbeing, health & safety
- Displays commitment through actively supporting all safety and wellbeing initiatives.
- Ensures own and others safety at all times.
- Complies with relevant safety and wellbeing policies, procedures, safe systems of work and event reporting.
- Reports all incidents/accidents, including near misses in a timely fashion.
- Is involved in health and safety through participation and consultation.
Tō tūranga i roto i te Manatū – Your place in the Ministry
The Data Engineer position reports into the Manager Risk Analytics and Data Science within the Immigration Risk and Border branch. The branch sits within the Immigration New Zealand group.
To mātou aronga – What we do for Aotearoa New Zealand
Hīkina Whakatutuki is the te reo Māori name for the Ministry of Business, Innovation and Employment. Hīkina means to uplift. Whakatutuki means to move forward, to make successful. Our name speaks to our purpose, Grow Aotearoa New Zealand for All.
To Grow Aotearoa New Zealand for All, we put people at the heart of our mahi. Based on the principles of Te Tiriti o Waitangi / The Treaty of Waitangi, we are committed to upholding authentic partnerships with Māori.
As agile public service leaders, we use our breadth and experience to navigate the ever-changing world. We are service providers, policy makers, investors and regulators. We engage with diverse communities, businesses and regions. Our work touches on the daily lives of New Zealanders. We grow opportunities (Puāwai), guard and protect (Kaihāpai) and innovate and navigate towards a better future (Auaha).
Ngā matatau – Our competencies
Cultivates innovation We create new and better ways for the organisation to be successful by challenging the status quo generating new and creative ideas and translating them into workable solutions.
Nimble learning We are curious and actively learn through experimentation when tackling new problems by learning as we go when facing new situations and challenges.
Customer focus We build strong customer relationships and deliver customer-centric solutions by listening and gaining insights into the needs of the communities we serve and actively seeking and responding to feedback.
Decision quality We make quality and timely decisions that shape the future for our communities and keep the organisation moving forward by relying on an appropriate mix of analysis, wisdom, experience, and judgement to make valid and reliable decisions.
Action oriented We step up, taking on new opportunities and tough challenges with purpose, urgency and discipline by taking responsibility, ownership and action on challenges, and being accountable for the results.
Collaborates We connect, working together to build partnerships with our communities, working collaboratively to meet shared objectives by gaining trust and support of others; actively seeking the views, experiences, and opinions of others and by working co-operatively with others across MBIE, the public sector and external stakeholder groups.
Te Tiriti o Waitangi
As an agency of the public service, MBIE has a responsibility to contribute to the Crown meeting its obligations under Te Tiriti o Waitangi (Te Tiriti). Meeting our commitment to Te Tiriti will contribute towards us realising the overall aims of Te Ara Amiorangi – Our Path, Our Direction, and achieve the outcome of Growing New Zealand for All. The principles of Te Tiriti - including partnership, good faith, and active protection – are at the core of our work. MBIE is committed to delivering on our obligations as a Treaty partner with authenticity and integrity and to enable Māori interests. We are committed to ensuring that MBIE is well placed to meet our obligations under the Public Service Act 2020 (Te Ao Tūmatanui) to support the Crown in strengthening the Māori/Crown Relationship under the Treaty and to build MBIE’s capability, capacity and cultural intelligence to deliver this.
Mahi i roto i te Ratonga Tūmatanui – Working in the public service
Ka mahitahi mātou o te ratonga tūmatanui kia hei painga mō ngā tāngata o Aotearoa i āianei, ā, hei ngā rā ki tua hoki. He kawenga tino whaitake tā mātou hei tautoko i te Karauna i runga i āna hononga ki a ngāi Māori i raro i te Tiriti o Waitangi. Ka tautoko mātou i te kāwanatanga manapori. Ka whakakotahingia mātou e te wairua whakarato ki ō mātou hapori, ā, e arahina ana mātou e ngā mātāpono me ngā tikanga matua o te ratonga tūmatanui i roto i ā mātou mahi.
In the public service we work collectively to make a meaningful difference for New Zealanders now and in the future. We have an important role in supporting the Crown in its relationships with Māori under the Treaty of Waitangi. We support democratic government. We are unified by a spirit of service to our communities and guided by the core principles and values of the public service in our work.
What does it mean to work in Aotearoa New Zealand’s Public Service?(external link) — Te Kawa Mataaho The Public Service Commission
