Senior Lead Data Software Engineer Python Spark Azure

We are seeking a Senior/Lead Data Software Engineer to join our team working on a scalable, ML-ready platform that enhances portfolio model development and deployment with advanced data governance and AI capabilities. You will play a key role in migrating from an IaaS Big Data platform to Azure-native Databricks, optimizing data workflows and improving data quality. Join us to contribute to innovative solutions that boost client services and regulatory compliance. Responsibilities Migrate and optimize over 500 data jobs using Azure Databricks optimization techniquesManage and process 12 TB of data efficiently across platformsTune machine learning models for Azure environments using Java Spark and Delta tablesUpdate and maintain libraries to address security vulnerabilitiesDevelop and maintain ETL/ELT pipelines using PySpark and related technologiesCollaborate with cross-functional teams to integrate GenAI capabilities into data workflowsMonitor data quality and implement improvements to ensure accuracy and reliabilityAutomate deployment and operational tasks using Terraform and GitLab CI/CDSupport data governance initiatives to comply with regulatory standardsTroubleshoot and resolve performance issues in data processing systemsDocument system processes and provide technical guidance to junior engineersImplement best practices for code quality and data securityParticipate in code reviews and knowledge sharing sessionsOptimize costs associated with data storage and processing Requirements Proficiency in Python and Spark with at least 3 years in data engineering rolesStrong experience with Azure Databricks and PySpark Proven expertise in designing and implementing ETL/ELT solutionsExperience migrating big data platforms to Azure-native servicesProficiency with Delta tables for model tuningKnowledge of data governance and regulatory compliance frameworksFamiliarity with Docker, Kubernetes (AKS), and Terraform for infrastructure automationAbility to manage large data volumes with high efficiencyExcellent problem-solving and analytical skillsStrong communication and collaboration abilitiesEnglish proficiency at B2 level or higher We offer We gather like-minded people:Top tech minds driving innovation in AI, cloud and digital platform modernizationSupportive team and agile, startup-like cultureHybrid by design mode and opportunity to work remotely within PolandChance to work abroad for up to 60 days annuallyBusiness-driven relocation opportunitiesWe provide growth opportunities:Career development programsThought leadership, mentoring, soft skills and well-being programsCertification (Anthropic, Gemini, GCP, Azure, AWS)English classesWe cover it all:Stable payParticipation in the Employee Stock Purchase Plan with a 15% discountBenefits package (health insurance, multisport, shopping vouchers)Referral bonuses up to $2,000Offices featuring entertainment and relaxation zones, table tennis and football, free snacks, coffee and moreCorporate, social and well-being eventsPlease, note:Benefits listed above are available to employees onlyWe are open for working with Contractors. Terms of B2B cooperation agreements are agreed individuallyWe will reach out to selected candidates exclusively EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.