DATA TRANSFER VALIDATION FLASK WEB SYSTEM
Descrição da oferta de emprego
The system will consist of multiple containers and facilitate the automated movement of data from SQL Server to Elasticsearch through Logstash, with a web interface for validation.
Project Scope.
Containerized Environment Setup.
Utilize the provided [login to view URL] file to build and manage the following containers.
SQL Server container (populated with provided SQL scripts) Logstash container configured with [login to view URL] Elasticsearch and Kibana containers Python container Data Population & Transfer.
Populate the SQL Server database automatically using the provided [login to view URL] and [login to view URL] scripts.
Transfer data from SQL Server to Elasticsearch using the provided Logstash configuration hwam.
onf.
Flask Web Application Development.
Build a Flask web application to serve as a validation interface.
The web interface should.
Automatically parse a provided Logstash pipeline configuration ([login to view URL]).
Connect to the SQL Server instance and list all data fields that are processed according to the configuration.
Connect to Elasticsearch using parameters specified in the configuration.
Validate data transfer by comparing SQL Server data points with their corresponding entries in Elasticsearch.
Deliverables.
Fully functioning Docker environment with all containers correctly communicating and operational.
Flask web interface with the capabilities listed above, ready to accept user input for different Logstash pipeline configurations.
Documentation on the setup, configuration, and use of the system, including how to test data validation through the web interface.
Requirements.
Proficiency in Docker and container orchestration Strong knowledge of Python and Flask Experience with Logstash pipelines, Elasticsearch, and Kibana Familiarity with SQL Server and data querying Ability to create an intuitive and user-friendly web interface for data validation Files Provided.
[login to view URL] [login to view URL] [login to view URL] [login to view URL] (Logstash configuration) [login to view URL] [login to view URL] python_app folder SQL scripts ([login to view URL] and [login to view URL]) Python Elasticsearch Containerization ID do Projeto.
# Sobre o projeto 14 propostas Aberto para ofertas Projeto remoto Ativo em 1 minuto atrás
Detalhes da oferta
- Indeterminado
- Em todo Portugal
- Indeterminado - Indeterminado
- 18/11/2024
- 16/02/2025
Fórum selecção is looking for a senior data engineer (m/f) for a corporate bank main activities: - design high-performance data pipelines (etl) to feed a dwh using state of the art data engineering techniques according to business specifications; - implement flow automation through ci/cd pipelines and......
Mínimo de 2 anos de experiência em desenvolvimento web frontend... descrição: trabalhar em projetos desafiadores, para o mercado internacional, desenvolvendo interfaces interativas, disruptivas e de alta performance que definem o futuro da web... na techframe, estamos à caça de um frontend web developer......
Fatores preferenciais: experiência em desenvolvimento de web services... transforme o futuro da web com a techframe! está pronto para desafiar o status quo e construir sistemas que vão além dos limites convencionais? a techframe, na vanguarda da inovação tecnológica, está à procura de um backend web......
Responsabilidades do cargoverificar e atualizar os cupões no sítio web https://cupaotemuportugal... com/ ;verificar a existência de novos cupões nos sítios web dos nossos parceiros e adicioná-los ao sítio web;análise de mercado: pesquisa de mercado e de concorrentes, identificação de tendências, recolha......
Net: mvc, web api, entity framework core; webservices (soap, rest, xml); mvc or web api; react... we also inform you that your data will be kept in the company for a period of 5 years... js (3+ years) experience with web applications (backend), distributed systems and apis preferred additional skills......
✅ competitive salary and remuneration system (incl... 10 (9th october)• fixed term, renewable full time employment contract for 6 months (according to the portuguese law)• we offer fully, 100% remote work system all over portugal (mainland)• shifts system: monday -sunday (7:00 – 21:00)-40h/5days per......
@confidentialnote: mne library of python will be used to explore, visualise and analysehuman neurophysiological data... ● clinic needs to click on “stop data stream” to stop the recording, after thesession is completed... ● once the board is set up, the clinic needs to click on “start data stream” tostart......
As a candidate you’ve decided to provide us your personal data... who we are: unikystem is a low-code bpm automation platform boosted by cognitive data capture with 100% accuracy, that turns any unstructured text data into business-critical information... required skills and experience: being a big......
Responsabilidades: levar a cabo projetos no âmbito do desenvolvimento da oferta para os clientes, através de soluções de business intelligence / data analytics; propor e implementar melhorias nas plataformas e processos existentes; envolvimento direto em projetos estratégicos que visam desenvolver soluções......
Análise de requisitos: colaborar com as partes interessadas para entender os requisitos do sistema e traduzi-los em especificações técnicas e planos de projeto... a familiaridade com frameworks e bibliotecas populares também é útil... desenvolvimento web: se o trabalho envolve desenvolvimento web, é......
