Test Engineer - Cloud Data Engineering

Job Description

About Scrumconnect

Scrumconnect Consulting is a UK-based digital transformation consultancy delivering agile, secure technology solutions for public and private sector clients. This is a fully remote role based in India, supporting our UK operations and client base. You will work closely with our UK leadership team and must be comfortable operating within UK regulatory and legal frameworks.

About The Role

Test Engineer on a large-scale cloud data engineering programme, operating across a modern AWS-native technology stack including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, and DynamoDB. You will own testing across automated pipelines, data workflows, and cloud infrastructure — identifying risks, championing test frameworks, and coaching colleagues in quality engineering practices.

This is a hands-on, technically deep role. You will write code, build and extend test frameworks, and take accountability for test quality across releases — while actively mentoring more junior team members in best practices.

Key Responsibilities

Automated testing

Design, implement, and maintain automated test suites across data pipelines, cloud services, and application layers. Ensure reproducibility and reliability across all releases.

Risk identification

Identify and raise awareness of risks arising from automated test results. Translate technical test findings into clear, prioritised risk communication for the team and stakeholders.

Framework development

Research, evaluate, and implement new test frameworks suited to the AWS data stack. Continuously improve existing frameworks to increase coverage and confidence in test outcomes.

Pipeline & data testing

Validate data pipelines built on Apache Airflow, AWS Glue, Athena, and EMR — including data integrity, transformation accuracy, and performance under load.

Mentoring & coaching

Mentor and advise team members in testing practices. Share knowledge actively, contribute to a quality-first culture, and support less experienced colleagues in developing test skills.

CI/CD integration

Integrate and manage tests within GitLab CI/CD pipelines. Ensure tests run automatically on commit, providing fast feedback loops and blocking regressions before deployment.

Technology stack

Apache AirflowAmazon AthenaAWS S3AWS GlueAWS EMRAWS EC2AWS ECRAWS DynamoDBAWS CloudWatchAWS IAMPythonJavaSQLBashGitLab CI/CDJupyter Notebooks

Skills And Experience Required

• Demonstrable experience as a Test Engineer or QA Engineer working on data pipelines or cloud-native environments
• Strong Python proficiency — used for test scripting, data validation, and automation in a production context

Interested in remote work opportunities in QA & Testing? Discover QA & Testing Remote Jobs featuring exclusive positions from top companies that offer flexible work arrangements.

• Working knowledge of Java and SQL for test development and data querying
• Scripting experience in Bash or equivalent for pipeline and environment automation
• Hands-on experience with the AWS ecosystem — particularly S3, Glue, Athena, EMR, CloudWatch, IAM, and DynamoDB
• Experience configuring and running tests within GitLab CI/CD pipelines
• Proven ability to design and implement automated test frameworks from scratch or extend existing ones
• Experience testing data pipelines — including data quality, integrity, transformation logic, and performance
• Ability to identify, articulate, and communicate testing risks clearly to both technical and non-technical audiences
• SFIA Level 4 capability — works autonomously, takes ownership of test quality, and influences others
• Experience mentoring or coaching junior engineers in testing practices
• Active SC clearance — mandatory at point of application, no exceptions
  
  

Diversity and Inclusion

At Scrumconnect Consulting, we believe that diversity drives innovation. We are committed to creating an inclusive environment where every individual is respected, valued, and supported. We welcome applications from candidates of all backgrounds and experiences, and we actively encourage applications from women, people with disabilities, under-represented communities, and those seeking flexible working arrangements.

About The Role

Test Engineer on a large-scale cloud data engineering programme, operating across a modern AWS-native technology stack including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, and DynamoDB. You will own testing across automated pipelines, data workflows, and cloud infrastructure — identifying risks, championing test frameworks, and coaching colleagues in quality engineering practices.

This is a hands-on, technically deep role. You will write code, build and extend test frameworks, and take accountability for test quality across releases — while actively mentoring more junior team members in best practices.

Key responsibilities

Automated testing

Design, implement, and maintain automated test suites across data pipelines, cloud services, and application layers. Ensure reproducibility and reliability across all releases.

Risk identification

Identify and raise awareness of risks arising from automated test results. Translate technical test findings into clear, prioritised risk communication for the team and stakeholders.

Framework development

Research, evaluate, and implement new test frameworks suited to the AWS data stack. Continuously improve existing frameworks to increase coverage and confidence in test outcomes.

Pipeline & data testing

Validate data pipelines built on Apache Airflow, AWS Glue, Athena, and EMR — including data integrity, transformation accuracy, and performance under load.

Mentoring & coaching

Mentor and advise team members in testing practices. Share knowledge actively, contribute to a quality-first culture, and support less experienced colleagues in developing test skills.

CI/CD integration

Integrate and manage tests within GitLab CI/CD pipelines. Ensure tests run automatically on commit, providing fast feedback loops and blocking regressions before deployment.

Technology stack

Apache AirflowAmazon AthenaAWS S3AWS GlueAWS EMRAWS EC2AWS ECRAWS DynamoDBAWS CloudWatchAWS IAMPythonJavaSQLBashGitLab CI/CDJupyter Notebooks

Skills And Experience Required

• Demonstrable experience as a Test Engineer or QA Engineer working on data pipelines or cloud-native environments
• Strong Python proficiency — used for test scripting, data validation, and automation in a production context
• Working knowledge of Java and SQL for test development and data querying
• Scripting experience in Bash or equivalent for pipeline and environment automation
• Hands-on experience with the AWS ecosystem — particularly S3, Glue, Athena, EMR, CloudWatch, IAM, and DynamoDB
• Experience configuring and running tests within GitLab CI/CD pipelines
• Proven ability to design and implement automated test frameworks from scratch or extend existing ones
• Experience testing data pipelines — including data quality, integrity, transformation logic, and performance
• Ability to identify, articulate, and communicate testing risks clearly to both technical and non-technical audiences
• SFIA Level 4 capability — works autonomously, takes ownership of test quality, and influences others
• Experience mentoring or coaching junior engineers in testing practices
• Active SC clearance — mandatory at point of application, no exceptions
  
  

check(event) ; career-website-detail-template-2 => apply(record.id,meta)" mousedown="lyte-button => check(event)" final-style="background-color:#68B54C;border-color:#68B54C;color:white;" final-class="lyte-button lyteBackgroundColorBtn lyteSuccess" lyte-rendered="">