Big Data Data Engineer (Spark / PySpark / Azure Databricks)

Experience: Minimum 3 years

• Demonstrable experience of at least 3 years in Data Engineering, Big Data, or the development of distributed processing solutions.
• Solid experience with Apache Spark and PySpark.
• Advanced knowledge of Spark optimization:
• Partitioning and shuffles.
• Persistence and cache.
• DAG and stages.
• Memory management.
• Execution and performance optimization.
• Proficiency in Python applied to data handling and processing.
• Experience with PySpark, Pandas, and NumPy.
• Experience in the design and development of ETL/ELT processes, data pipelines, and datasets.
• Experience with Azure Databricks and Delta Lake.
• Knowledge of columnar data processing with Pandas UDF, Vectorized UDF, and Apache Arrow.
• Experience or knowledge in distributed and iterative algorithms.
• Practical use of AI tools for software development, such as GitHub Copilot, GenAI, or LLMs, aimed at streamlining reviews, testing, and validations.
• Ability to communicate in a technical, clear, and effective manner with clients and multidisciplinary teams.

• Knowledge of Approximate Computing: HyperLogLog, Bloom Filters, sketching, and streaming data statistics.
• Experience in cost and performance optimization of workloads in Azure Databricks.
• Knowledge of Lakehouse architectures and data management with Delta Lake.
• Experience working in Agile environments.

• Permanent contract.
• Competitive salary, in accordance with the experience and knowledge provided.
• Opportunities for professional development and continuous training.
• Participation in projects of high technological impact.
• Collaborative and innovative working environment.
• Flexible hours and working modality to be defined.

At Métrica España, we are looking for a Data Engineer with experience in Big Data environments, distributed processing, and workload optimization using Apache Spark, PySpark, and Azure Databricks.

We are looking for a proactive, autonomous, and decisive person, with a solid technical foundation and the ability to work directly with the client. If you are interested in designing efficient, scalable, and robust data solutions, and using AI tools to streamline development, this is your opportunity!

• Design, development, and maintenance of distributed data processes with Apache Spark and PySpark.
• Development and optimization of ETL/ELT pipelines and design of scalable datasets.
• Analysis and improvement of Spark process performance: partitioning, shuffles, persistence, cache, DAG execution, stages, and memory management.
• Development of columnar processing solutions using Pandas UDF, Vectorized UDF, and Apache Arrow.
• Development and maintenance of solutions in Azure Databricks and Delta Lake.
• Optimization of workloads, execution times, resource consumption, and costs associated with data processes.
• Application of distributed and iterative algorithms for processing large volumes of information.
• Use of Approximate Computing techniques, such as HyperLogLog, Bloom Filters, sketching, and streaming statistics, when necessary.
• Use of AI tools applied to development —GenAI, Copilot, and LLMs— for code review, test generation, validation, and productivity improvement.
• Direct collaboration with the client and technical teams for requirement gathering, solution definition, and communication of technical progress.
• Technical documentation of developments, pipelines, and implemented processes.
• Identification and proposal of continuous improvements in quality, efficiency, performance, and robustness.

Modality: 100% remote

SBA: 34,000€