Big Data Developer

About the Сompany

Nexus Cognitive are innovators in the world of data and analytics, dedicated to unlocking the full potential of their customer’s data. NexusOne is their tech-enabled managed service for big data infrastructure that combines cutting-edge technology with deep industry expertise to deliver comprehensive, tailored solutions.

Their mission is to enable your business to harness the power of big data, transforming complex data sets into actionable insights that disrupt your industry and deliver a competitive advantage.  They handle the heavy lifting of big data management and governance so their customers can focus on making informed, strategic decisions that drive growth and innovation.

Role Summary

The Big Data Developer plays a key role in the modernization of data ecosystem — supporting the migration of legacy MAPR/Cloudera/Hortonworks applications to open-source frameworks compatible with NexusOne.

This individual will focus on refactoring, optimizing, and validating data processing pipelines, ensuring performance, scalability, and alignment with enterprise data standards.
The role requires strong technical expertise across distributed data systems, open-source frameworks, and hybrid data environments.

Core Responsibilities

• Analyze, refactor, and modernize Spark/MapReduce/Hive/Tez jobs for execution within NexusOne’s managed Spark and Trino environments.
• Design, build, and optimize batch and streaming pipelines using Spark, NiFi, and Kafka.
• Convert existing ETL jobs and DAGs from Cloudera/MAPR ecosystems to open-source equivalents.
• Collaborate with Data Engineers and Architects to define new data ingestion and transformation patterns.
• Tune performance across large-scale data processing workloads (partitioning, caching, resource allocation).
• Implement data quality and validation frameworks to ensure consistency during migration.
• Support code reviews, performance tests, and production readiness validation for migrated workloads.
• Document conversion approaches, dependencies, and operational runbooks.
• Partner with Wells Fargo application SMEs to ensure domain alignment and
  business continuity.

Must-Have Core Skills

• Core Frameworks: Apache Spark, PySpark, Airflow, NiFi, Kafka, Hive, Iceberg, Oozie
• Programming Languages: Python, Scala, Java
  
• Data Formats & Storage: Parquet, ORC, Avro, S3, HDFS
• Orchestration & Workflow: Airflow, DBT
• Performance Optimization: Spark tuning, partitioning strategies, caching, YARN/K8s resource tuning
• Testing & Validation: Great Expectations, Deequ, SQL-based QA frameworks
• Observability & Monitoring: Datadog, Grafana, Prometheus

Ideal Background

• 4–8 years of experience in big data engineering or application modernization in enterprise settings.
• Prior experience with Cloudera, MAPR, or Hadoop ecosystems, transitioning to open-source frameworks.
• Strong understanding of distributed data architectures and data lake design principles.
• Exposure to hybrid or cloud-native environments (AWS, GCP, or Azure).
• Familiarity with regulated environments (financial services, telecom, healthcare) is a plus.
• Advanced English proficiency (B2–C1 or higher).

Success Criteria

• Successful refactoring and execution of legacy data pipelines within NexusOne environments.
• Measurable performance improvements (execution time, cost optimization, data quality metrics).
• Delivered migration artifacts — including conversion patterns, reusable scripts, and playbooks.
• Positive feedback from Wells Fargo application owners on migration support and knowledge transfer.
• Consistent adherence to coding standards, documentation, and change management practices.

We offer

• We welcome new ideas and allow you to make an immediate impact on the team.
• Flexible Paid time off (PTO for any reason, including sick days (no specified limits) and flexible work schedule.
• Personal laptop.
• Health/Sport Budget.
• Fully remote.