Ir à oferta completa

PT | DATA ENGINEER_L

Lisboa - Lisboa

Descrição da oferta de emprego

Description.
Position Summary.
We are looking for a Databricks Specialist Consultant with a deep focus on Unity Catalog and Microsoft Purview, responsible for migrating files in Parquet format stored in Azure Storage Account containers to Delta Table format, with the implementation of a layered architecture (Bronze, Silver, Gold) in Databricks.
The consultant will also be responsible for integrating these Delta Tables into Unity Catalog with centralised governance via Microsoft Purview, ensuring that the data is accessible and well governed while being used for dynamic reporting in Power BI.
Responsibilities.
- Expertise in Unity Catalog.
- Migrate Parquet files stored in Azure Storage Account to Delta Tables registered in Unity Catalog in Databricks.
- Apply granular access control at table, column and schema level using RBAC in Unity Catalog, ensuring compliance and security.
- Configure and optimise Unity Catalog to provide centralised governance over all data in Databricks, ensuring that permissions and data lineage are clearly defined.
- Advanced integration with Microsoft Purview.
- Integrate Unity Catalog with Microsoft Purview for automatic cataloguing, data lineage tracking, and auditing.
- Ensure that all data changes, permissions and metadata are visible and auditable via Purview, guaranteeing compliance with regulations such as GDPR and HIPAA.
- Implementation of Layered Data Architecture.
- Implement Bronze, Silver, Gold architecture in Databricks, with different layers of data for ingestion, transformation and final exposure for reporting.
- Create Databricks clusters suitable for each layer, optimising performance and guaranteeing scalability and security in data processing.
- Notebook Conversion and Delta Table Optimisation.
- Review and migrate existing notebooks that handle Parquet files to use Delta Tables registered in Unity Catalog.
- Implement performance optimisations in Delta Tables, using commands such as OPTIMIZE and VACUUM to improve query efficiency and free up space.
- Reports and Visualizations with Power BI.
- Ensure that data transformed and governed via Delta Tables in Databricks is accessible for real-time reporting in Power BI, using Direct Query to ensure data is always up-to-date.
Technical skills required.
- Expert in Unity Catalog.
- Advanced experience in configuring, managing and optimizing Unity Catalog in Databricks, including access control, security policies and governance.
- Microsoft Purview.
- Proficiency in Microsoft Purview, with experience in integrating and maintaining data governance with Unity Catalog, ensuring traceability, auditing and compliance.
- Databricks and Delta Lake.
- Solid experience using Databricks for large-scale data manipulation, especially utilizing Delta Lake and Delta Tables.
- Ability to implement and optimize data pipelines in Bronze, Silver and Gold tiers in Databricks.
- Azure Storage.
- In-depth knowledge of Azure Storage Accounts, Azure Blob Storage, and Azure Data Lake Storage (ADLS), including the manipulation of Parquet files for storage and performance optimization.
- Power BI.
- Ability to integrate Databricks data into Power BI to create dynamic dashboards and reports, ensuring security permissions are respected.
Desirable Skills.
- Data Governance and Security.
Deep understanding of data governance, compliance policies, and security practices, especially in the context of sensitive data.
- Pipeline Automation.
Experience in automating and orchestrating data pipelines in an Azure environment, with a focus on efficiency and resource optimization.
Mandatory Requirements.
- Expertise in Unity Catalog in Databricks.
- Advanced experience with Microsoft Purview for data governance and auditing.
- Proven ability to migrate and optimize data in Delta Tables and register in Unity Catalog.
- In-depth knowledge of data manipulation in Azure Storage Accounts and Databricks.
- Experience with data integration in Power BI for dynamic reports and visualizations.
Nice to Have.
- Microsoft Azure certifications, such as Azure Data Engineer or Azure Solutions Architect.
- Experience with PySpark for task automation and optimization in Databricks.
Work Location.
Lisbon 60% Remote and 40% Presential
Ir à oferta completa

Detalhes da oferta

Empresa
  • Alium Consultancy Ltd
Localidade
Endereço
  • Indeterminado - Indeterminado
Tipo de Contrato
  • Indeterminado
Data de publicação
  • 22/11/2024
Data de expiração
  • 20/02/2025
Senior Data Engineer
FÓRUM SELECCAO

Fórum selecção is looking for a senior data engineer (m/f) for a corporate bank main activities: - design high-performance data pipelines (etl) to feed a dwh using state of the art data engineering techniques according to business specifications; - implement flow automation through ci/cd pipelines and......

Mechanical Engineer (m/f) | Lisboa
Claire Joster by Selection

Function as a mechanical engineer, you will be responsible for: technical support on the preparation of proposals with a focus on rotating and static equipment scope of work; perform engineering activities from conceptual, feed, and detail design; develop materials lists, data sheets, specifications......

Automation and Robotics Engineer
TECNICOAT, LDA

Familiarity with plc programming and industrial automation protocols... with a commitment to excellence and a passion for pushing boundaries, we are seeking a highly skilled and motivated automation and robotics engineer to join our talented team... position overview: we are looking for a creative automation......

Senior QA Engineer
Equação it

);• user acceptance testing knowledge;• cypress;• browserstack;• java;• synthetic data creation;• iam test accounts;• appium;• experience with ci/cd pipelines with tools such jenkins, ado, or github actions;• knowledge in qa best practices and standards;• coach and mentoring;• audits;• sql server;• tosca......

Test Engineer (Automation) – Phyton
Newin

· coordinate with program and development management teams in product development lifecycle to conform end user product and quality requirements and shipment schedule... · answer technical issues to resolve and follow up... · proactively identify additional test automation opportunities and analyse potential......

O SEVEN está a recrutar INSTRUTORES MUSCULAÇAO E PT
SEVEN FITNESS CLUB

Seven fitness club - estamos a recrutar personal trainers - part time e full time (m/f) quais as tuas funções? - avaliações físicas - horas fixas na sala de musculação - prescrição de treino individualizado valorizamos: - atitude comercial - proatividade e sentido de organização - flexibilidade de horário......

O seven está a recrutar instrutores musculaçao e pt
Seven fitness club

Matosinhos seven fitness club - estamos a recrutar personal trainers - part time e full time (m/f) quais as tuas funções? - avaliações físicas - horas fixas na sala de musculação - prescrição de treino individualizado valorizamos: - atitude comercial - proatividade e sentido de organização - flexibilidade......

Agente de Reservas PT ENG
Minor Hotels Portugal

Requisitos do trabalho perfil- formação em hotelaria, turismo ou similar (fator preferencial);- competências de venda (experiência em gestão de reservas ou em área comercial será valorizada);- experiência em atendimento ao cliente (fator preferencial);- excelente capacidade de comunicação;- fluente em......

DevOps Engineer
Newin

Obligations: openshift/kubernetesdockersgithubartifact (jfrog)ansiblepython/powershelljira/confluencelinux administrationbasic knowledge of databasesteam player and being able to be part of a community to share information/knowledge good to have: azure/awsgerman speaker job type:full time......

Gerente de Compras Russo Pt
Relato Deslumbrante

Vaga: gerente de compras para rede de lojasrequisitos: proatividade e desejo de crescimento profissional pensamento analítico conhecimento de russo e português (apenas esses idiomas) responsabilidades: análise de mercado negociação e fechamento de contratos elaboração de relatórios......