We are looking for a Senior Data Engineer with minimum 5+ years of professional experience to design, build, and maintain scalable, high-performance data platforms that power our analytics and machine learning initiatives. You will work in a distributed, data-intensive environment and collaborate with cross-functional teams to ensure data accuracy, reliability, and security.
Design, develop, and maintain scalable data processing pipelines and workflows using frameworks such as Apache Spark, PySpark, and Apache Beam.
Build and maintain microservices in Python that serve data-driven features in production.
Develop internal tools to support CI/CD pipelines, experiment tracking, and data versioning.
Collect, process, and integrate large datasets from multiple sources, including databases, file systems, and APIs.
Ensure data integrity, consistency, and quality through robust validation and monitoring processes.
Optimize data systems for performance, scalability, and high availability.
Implement best practices for data security, access control, and privacy.
Collaborate with data scientists, analysts, and engineers to support analytics and ML workflows.
Lead complex migration initiatives involving the transition from on-premise Cloudera (HDFS, Hive, Impala, HBase) environments to cloud-native platforms (Snowflake or Databricks), ensuring zero data loss and minimal downtime.
Architect and build a large-scale, greenfield data platform from the ground up, including the setup of end-to-end infrastructure, data ingestion layers, transformation frameworks, and high-throughput integrations with Snowflake or Databricks.
Strong understanding of distributed systems and modern data architectures.
Proven, hands-on experience with large-scale data platform migrations, specifically transitioning from Cloudera (CDH/HDP) to either Snowflake or Databricks. This includes re-architecting Hive/Impala workloads, migrating HDFS data, and refactoring legacy ETL processes to leverage cloud-native features (e.g., Snowpipe, Auto-ingestion, or Delta Live Tables).
Deep technical expertise in building and orchestrating a high-performance data platform from scratch, including the implementation of complex integrations (Reverse ETL, CDC, API ingestions) specifically targeting Snowflake (using Snowpipe Streaming, Tasks, Streams) or Databricks (using Auto Loader, Delta Sharing, Unity Catalog).
5+ years of professional experience in software engineering or data engineering.
Strong software engineering skills with Python in large-scale, high-performance production environments.
Hands-on experience with Spark/PySpark and other big data frameworks.
Opplane specializes in providing advanced data-focused solutions for financial services, telecommunication, and reg-tech to accelerate their digital transformation journey. Opplane leadership team is comprised of Silicon Valley serial entrepreneurs and experienced executives. Its expertise comes from years of specific industry experience at some of the world’s top companies, such as PayPal, Xerox Parc, Amazon, Wells Fargo, SoFi in the areas of product management, data technology, data governance, data privacy, security, machine learning, and risk management.
🌍 Global & Multicultural – Diverse perspectives, global collaboration (US, Portugal, India and Singapore offices)
⚡ Startup Energy – Fast-moving, impact-driven environment
💪 Ownership Mindset – Engineers own what they build
🤝 Collaborative & Friendly – Open, curious, and supportive culture