DATA ENGINEER (SENIOR) W/ PYSPARK
Descrição da oferta de emprego
In this role, you will be responsible for designing, building, and maintaining robust data pipelines and architectures that drive our data-driven decisions.
You will work closely with cross-functional teams to ensure data availability, quality, and accessibility for various business needs.
Key Responsibilities Design, implement, and optimize large-scale data pipelines using PySpark.
Manage and maintain data in SQL-based big data columnar databases such as Apache Druid, Google BigQuery, Cassandra, SAP HANA, etc.
Leverage Google Cloud Platform (GCP) services like BigQuery, Firestore, and Pub/Sub to support data infrastructure.
Develop and orchestrate data workflows using Airflow, particularly within the GCP environment (Cloud Composer).
Collaborate with data scientists, analysts, and other stakeholders to ensure the availability and reliability of data.
Optimize and troubleshoot data systems for performance, scalability, and reliability.
Document data processes, architectures, and pipelines for maintenance and scalability.
Qualifications Requirements PySpark.
Strong experience and proficiency in building and optimizing data pipelines.
SQL-based Big Data Columnar Databases.
Expertise in working with databases like Apache Druid, Google BigQuery, Cassandra, SAP HANA, etc.
Google Cloud Platform (GCP).
Hands-on experience with BigQuery, Firestore, Pub/Sub.
Airflow.
Proven experience with Airflow, specifically within GCP (Cloud Composer) for data orchestration.
Communication Skills.
Excellent verbal and written communication skills, with the ability to collaborate effectively across teams.
Scala.
(Nice to have) Familiarity or experience with Scala programming language.
Firebase.
(Nice to have) Experience with Firebase services.
Qualifications Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
Minimum of 5+ years of experience in data engineering or a related field.
Proven track record of working with large-scale data processing and distributed systems.
Strong problem-solving skills and ability to work independently in a fast-paced environment.
Additional Information If you are a passionate Data Engineer with a strong background in big data technologies and are eager to take on new challenges, we would love to hear from you! Apply now to join our team and contribute to cutting-edge data solutions.
I'm interested I'm interested Privacy Policy
Detalhes da oferta
- Alter Solutions
- Em todo Portugal
- Indeterminado - Indeterminado
- 28/08/2024
- 26/11/2024
We are looking for a senior qa engineer with the following requirements: requisitos do trabalho • minimum of +6 years of relevant experience;• repository practice (i... );• user acceptance testing knowledge;• cypress;• browserstack;• java;• synthetic data creation;• iam test accounts;• appium;• experience......
Maintenance) of raw data from various data sources into the data warehouse;• automation and optimization of data imports;• preparation and normalization of data;• creating sap bo reports with multiple data sources;• at least 3 year’s experience in data analysis and/or reporting, ideally in a data warehouse......
Devops engineer location: carnaxide, lisbon, portugal, occupation: full time min... com/careers/devops-engineer/ as a candidate you've decided to provide us your personal data... who we are: unikystem is a low-code bpm automation platform boosted by cognitive data capture with 100% accuracy, that turns......
Claire joster is currently recruiting for a portuguese company of power generating, oil&gas; and material handling sectors, which intends to strengthen its internal structure with the integration of a mechanical engineer (m/f) in lisbon... function as a mechanical engineer, you will be responsible for:......
Please include 'automation and robotics engineer application' in the subject line... position overview: we are looking for a creative automation and robotics engineer to contribute to the design, development, and implementation of cutting-edge automation and robotics solutions... collaborate with cross-functional......
Contamos contigo para te juntares à nossa família!envia o teu currículo para: *****@***** indicando no assunto do email a referência jobatus/sénior salesforcewill you step-in? requisitos do trabalho o que procuramos:- experiência e gosto pela vertente de analista funcional;- experiência na implementação......
Test engineer (automation) – air traffic management airspace industry in the role of the automation test engineer at atos you will: · develop and configure test automation frameworks and execute stability and performance tests... · analyse and verify best automated and manual test approaches and execute......
We are looking for a sap mm senior with the following requirements: requisitos do trabalho requeriments:• s/4hana implementations... fluent in english;• +8 anos exp; outros dados de posição • duration: contract for 12 months (minimum);• hybrid, lisbon... descrição:equação it is a company focused on boosting......
Job missionbiosurfit is looking for a talented engineer to join its research and development team... biosurfit is a portuguese diagnostics company focused on the development and manufacture of ivd tests for the point-of-care market with highly innovative proprietary technology and dedicated to the provision......
We are looking for a sap hcm senior with the following requirements: requisitos do trabalho • more than 5 years of experience in sap hcm modules: py;• official language: english;• flex work (base lisbon);preferential requests:• pa, time management, om and cats;• portuguese language; outros dados de posição......
