Data Engineer Job at Halian
Responsible to collect, curate and catalogue data. Data will be loaded into the Hadoop data store from source systems or external sources and will be profiled and catalogued. Data will be mapped and transformed into the SAP Financial Services Data Model in the SAP Hana layer. Data will then be published into data marts or operational systems or to analytical users.
• Implement identified data ingestion patterns, using Informatica BDM or other approved technology.
• Technically profile the data for technical issues and potentially correct those issues.
• Produce profiling reports to be provided to stakeholders.
• Catalogue the data and load metadata into the Data Catalogue.
• Where required perform data enrichment and improvement
• Create data transformation logic using Informatica BDM
• Consume data from Kafka queues and publish data to kafka queues
• Where required create associations or relationships between data elements within the catalogue or in the Hadoop data layer
• Perform Data Quality assessment based on quality rules defined by data owner /stewards
• Responsible for providing data and information that is accurate, reliable and congruent, and ensure the information is easily available for direct consumption or integration with other systems.
• Preferably currently based in Dubai and available immediately
• Minimum 1 years of professional experience in Banking & Finance industry
• Minimum 2 years professional experience in Data Warehouse
• Knowledge/exposure to Modern Data Warehouse Architecture
• Agile working practices
• 2+ years of working experience with ETL tool, preferably Informatica
• Solid knowledge of Data Warehouse principles
• Experience of working with Hadoop
• Experience of working with Hadoop related tools e.g. Spark, Sqoop, Kafka, Python, Scala
• Experience of working in Agile teams as part of a squad.
• Experience working in Banking or Finance Industry
• Working knowledge in technical aspects of Data Governance including glossary, metadata, data quality, master data management & data lifecycle management.
Founded in 1996, Halian is an IT services company with a strong track record of delivering value for our customers across the UK, Europe, The Middle East and Africa.