Data Engineer
- Ireland
- Permanent
- Full-time
We are seeking a skilled Data Engineer to join our client's product engineering team, supporting Business Intelligence and Data Science initiatives primarily using AWS technologies. You will collaborate closely with their corporate technology team to build, automate, and maintain AWS infrastructure.
This role requires a highly competent, detail-oriented individual who stays current with evolving data engineering technologies.
Key Responsibilities
- Collaborate with data consumers, producers, and compliance teams to define requirements for data solutions with executive-level impact.
- Design, build, and maintain solutions for Business Intelligence and Data Science on AWS, including:
- Data ingestion pipelines
- ETL/ELT processes (batch and streaming)
- Curated data products
- Integrations with third-party tools
- Support and enhance the data lake, enterprise data catalog, cloud data warehouse, and data processing infrastructure.
- Provision and manage AWS services and infrastructure as code using Amazon CDK.
- Provide input on product/vendor selection, technology strategies, and architectural design.
- Identify and implement improvements to reduce waste, complexity, and redundancy.
- Manage workload efficiently to meet service levels and KPIs.
- Execute incident, problem, and change management processes as required.
- Degree in Information Systems, Computer Science, Statistics, or a related quantitative field.
- 3+ years experience with Spark in production environments, handling batch and stream processing jobs. Exposure to Ray is advantageous.
- 3+ years experience with cloud data warehousing tools such as Snowflake, Redshift, BigQuery, or ClickHouse.
- Expert SQL skills, with exposure to HiveQL.
- Proficiency in Java, Scala, Python, and Typescript programming.
- Strong understanding of AWS security mechanisms, particularly relating to S3, Kinesis, EMR, Glue, and LakeFormation.
- Experience with GitHub, DataDog, and AWS.
- Proven ability to learn and apply open-source tools independently.
- Strong ownership mindset and proactive approach.