AWS Data Engineer (Expert Level)
- OpenSource Intelligent Solutions
- Midrand, Gauteng, South Africa
Job Description
Are you an expert in AWS Data Engineering with a passion for building scalable, efficient, and innovative data solutions? We are seeking a highly skilled AWS Data Engineer to join our dynamic team!
Key Responsibilities:
- Design, develop, and optimize data pipelines and workflows using AWS tools and services.
- Implement infrastructure as code using Terraform to manage and automate cloud resources.
- Develop and maintain complex data processing systems utilizing Python 3x and PySpark.
- Manage databases and write advanced SQL queries on Oracle and PostgreSQL.
- Work with big data technologies, containerization using Docker, and Unix/Linux environments.
- Collaborate with cross-functional teams using enterprise collaboration tools like Confluence and JIRA.
- Develop and maintain technical documentation, ensuring accurate data transformations and thorough testing.
- Leverage your knowledge of AWS components, including Glue, S3, Lambda, Kinesis, and DynamoDB to build robust data solutions.
- Enhance data quality using tools such as Great Expectations and ensure the accuracy of data processing.
- Stay updated on industry best practices and contribute to continuous improvement within the team.
Essential Skills and Qualifications:
- Above Average Experience/Understanding in:
- Terraform (Infrastructure as Code)
- Python 3x
- SQL (Oracle/PostgreSQL)
- PySpark
- Boto3
- ETL Processes
- Docker
- Linux/Unix
- Big Data
- PowerShell/Bash
- BMW Cloud Data Hub (CDH)
- BMW CDEC Blueprint
Additional Required Skills:
- Experience with data formats like Parquet, AVRO, JSON, XML, CSV.
- Basic networking knowledge and troubleshooting capabilities.
- Experience with REST APIs, data modelling, and software design patterns.
- Business Intelligence (BI) experience is a plus.
Qualifications:
- Relevant IT/Business/Engineering Degree.
- Preferred certifications: AWS Certified Cloud Practitioner, AWS Certified SysOps Associate, AWS Certified Developer Associate, AWS Certified Architect Associate, AWS Certified Architect Professional, HashiCorp Certified Terraform Associate.
Why Join Us?
- Innovative Environment: Be part of a cutting-edge team driving data solutions with the latest AWS technologies.
- Career Growth: We support continuous learning and offer opportunities for professional development.
- Collaborative Culture: Work with a team of experts in a fast-paced, agile environment.
- Impactful Work: Contribute to projects that have a significant impact on the business and our clients.
How to Apply:
If you have the expertise and passion for AWS Data Engineering, we'd love to hear from you! Please submit your resume and a cover letter detailing your relevant experience and why you're the perfect fit for this role.