We are looking for a Data Analytics Engineer / Data Engineer with at least 3 years of experience in data engineering and viz, capable of managing the entire data lifecycle, from web and mobile data collection to reporting via Power BI. The ideal candidate will be responsible for designing and maintaining scalable data pipelines, ensuring data quality, and delivering insights that drive strategic decisions.
Technical Responsibilities:
Data Pipeline Management: Oversee and orchestrate data pipelines from collection to analysis using Airflow.
Web and Mobile App Scraping: Maintain and improve our scraping tools utilizing Python, Selenium, requests, and Appium.
Data Collection: Gather online data through our scraping tools and develop new ones when needed.
Data Modeling: Design, create, and maintain data models using SQL.
Data Processing: Process and restructure data via ELT/ETL pipelines.
Data Cleaning Pipelines: Implement cleaning pipelines incorporating NER and LLM models.
Azure Resource Management: Manage resources on the Azure platform.
Team Supervision: Lead and mentor a team of data professionals.
Reporting and Dashboards: Develop and manage dashboards and reporting tools using Power BI.
Functional Responsibilities:
Client Workshops: Facilitate workshops to gather and understand client requirements.
Documentation: Draft detailed specifications and project charters.
Project Roadmaps: Define project roadmaps and timelines.
Task Prioritization: Prioritize tasks and projects effectively.
Technical Stack:
SQL
Microsoft Power BI
Microsoft Excel
Python
Linux
MySQL
Docker
DBT
Azure