LEAD - GCP & DATA ENGINEER
Descrição da oferta de emprego
Summary.
• 8-9+ years of Total Experience • 6+ years of direct hands-on experience with the GCP BigQuery, Apache Airflow/ GCP Cloud Composer • Extensive experience with PySpark.
Strong expertise in developing and optimizing large-scale data processing solutions using Apache Spark with Python.
• Big Data Ecosystem.
Proficiency with Hadoop, HDFS, Hive, YARN, and other big data technologies.
Detailed JD.
• 9+ years of direct hands-on experience with the GCP BigQuery, Apache Airflow/ GCP Cloud Composer • Extensive experience with PySpark.
Strong expertise in developing and optimizing large-scale data processing solutions using Apache Spark with Python.
• Big Data Ecosystem.
Proficiency with Hadoop, HDFS, Hive, YARN, and other big data technologies.
• Experience with data architecture.
Hands-on experience designing and managing scalable data architectures, including data lakes and data warehouses.
• Cloud platforms.
Deep knowledge of GoogleCloud Platform, particularly services like Apache Airflow,Cloud Composer, BigQuery • ETL pipeline development.
Expertise in designing, building, and managing ETL processes with PySpark.
• SQL expertise.
Advanced knowledge of Spark SQL for data transformations and optimization of complex queries.
• Experience in distributed systems.
In-depth understanding of distributed computing principles, fault tolerance, and data partitioning.
• DevOps practices.
Experience with containerization (Docker), orchestration (Kubernetes), and managing Spark workloads in containerized environments.
• Performance tuning.
Proven ability to optimize Spark jobs, including memory management, partitioning strategies, and cluster resource management.
• Version control and CI/CD.
Proficiency with Git, Jenkins, CI/CD tools to automate data pipeline deployment and management.
• Team leadership.
Experience leading teams, mentoring junior engineers, and fostering a collaborative environment.
Skills.
Analytical Thinking Google Cloud Platform Python Data Warehousing Hadoop Hive What you can expect from us.
Together, as owners, let’s turn meaningful insights into action.
Life at CGI is rooted in ownership, teamwork, respect and belonging.
Here, you’ll reach your full potential because… You are invited to be an owner from day 1 as we work together to bring our Dream to life.
That’s why we call ourselves CGI Partners rather than employees.
We benefit from our collective success and actively shape our company’s strategy and direction.
Your work creates value.
You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise.
You’ll shape your career by joining a company built to grow and last.
You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons.
Come join our team—one of the largest IT and business consulting services firms in the world.
Detalhes da oferta
- CGI
- Em todo Portugal
- Indeterminado - Indeterminado
- Indeterminado
- 11/12/2024
- 11/03/2025
Fórum selecção is looking for a senior data engineer (m/f) for a corporate bank main activities: - design high-performance data pipelines (etl) to feed a dwh using state of the art data engineering techniques according to business specifications; - implement flow automation through ci/cd pipelines and......
Claire joster is currently recruiting for a portuguese company of power generating, oil&gas; and material handling sectors, which intends to strengthen its internal structure with the integration of a mechanical engineer (m/f) in lisbon... function as a mechanical engineer, you will be responsible for:......
Please include 'automation and robotics engineer application' in the subject line... position overview: we are looking for a creative automation and robotics engineer to contribute to the design, development, and implementation of cutting-edge automation and robotics solutions... collaborate with cross-functional......
We are looking for a senior qa engineer with the following requirements: requisitos do trabalho • minimum of +6 years of relevant experience;• repository practice (i... );• user acceptance testing knowledge;• cypress;• browserstack;• java;• synthetic data creation;• iam test accounts;• appium;• experience......
Test engineer (automation) – air traffic management airspace industry in the role of the automation test engineer at atos you will: · develop and configure test automation frameworks and execute stability and performance tests... · execute performance testing and present results for validating and analysis......
Obligations: openshift/kubernetesdockersgithubartifact (jfrog)ansiblepython/powershelljira/confluencelinux administrationbasic knowledge of databasesteam player and being able to be part of a community to share information/knowledge good to have: azure/awsgerman speaker job type:full time......
Therefore we encourage all individuals to apply! please provide your cv (already with certificates, employer's references, etc... com/de/pixida3) apply now! pixida is proud to be an equal opportunity employer! we are committed to a work environment that supports, inspires and respects all individuals......
Minimo 6 anos de experiência* licenciatura em tecnologia ou engenharia informática (ou similar)* experiência em java* conhecimento de ferramentas de ci/cd (por exemplo, jenkins, circle ci)* experiência com testng: anotações e framework* experiência com bdd, nomeadamente cucumber* experiência prática......
Rainsteal oil and gas is a multidiscipline supplier to the oil and gas industry... vacant positions: roustabouts / roughnecks, welders / mechanics, rig operators / drillers, engineers (petroleum and mechanical) health and safety officer, superintendent smp, smp supervisor, document controller clerk,......
Administrative department business analyst, payroll manager, marketing specialist, administration supervisor, human resources officer, financial analyst, senior marketing analyst, logistics coordinator / expert, procurement officer, secretary / office assistants / office clerks / front desk clerks, account......
