Lead Big Data Engineer

Required skills

— 5+ years of experience in data engineering, with a focus on big data technologies.

— Advanced skills in Scala and/or Python.

— Proficiency in Apache Spark/PySpark/Spark SQL DSL

— Strong experience with AWS (EC2, IAM, S3, Glue, EMR)

— Extensive knowledge of CI/CD processes and tools (Jenkins, GitHub Actions).

— Proficiency in data processing pipelines/database design and architecture (Lambda, Kappa, Datalake/Lakehouse architecture)

— Familiarity with Databricks (Jobs, SQL Warehouse, OverWatch, Unity Catalog).

— Strong understanding of data streaming and real-time data processing.

— Experience with message brokers (Kafka) and Change Data Capture (CDC) techniques.

— Proficiency in SQL and NoSQL databases.

— Level of English: Upper-Intermediate.

As a plus

— Experience with Apache Airflow, including writing plugins and custom operators.

— Experience with Terraform

— Strong grasp of AWS data platform services and their strengths/weaknesses

— Strong experience using Jira, Slack, JetBrains IDEs, Git, GitLab, GitHub, Docker, Jenkins

Responsibilities

— Lead the design, development, and maintenance of scalable and efficient data pipelines.

Conduct code reviews and create design documents for new features.

— Provide technical support and mentorship to other teams within the organization.

— Participate in management meetings to discuss priorities, scopes, deadlines, and cross-team dependencies.

— Develop, optimize, and tune Apache Spark jobs (Scala, PySpark, Spark SQL).

— Implement CI/CD processes using Jenkins and GitHub Actions.

— Design and implement data processing pipelines using AWS and GCP services.

— Work with Apache Airflow to manage and automate workflows, including custom plugin development.

— Utilize Databricks for job scheduling, SQL warehousing, and data visualization.

We offer

— High compensation according to your technical skills;

— Interesting projects with great Customers;

— 5-day working week, 8-hour working day, flexible schedule;

— Democratic management style & friendly environment;

— Full remote;

— Annual Paid vacation — 30 b/days + unpaid vacation;

— Paid sick leaves — 6 b/days per year;

— Corporate Perks (external training, English courses, corporate events/team buildings);

— Professional and personal growth.