Data Engineer | Python + SQL + PostgreSQL

• Solid experience developing Python scripts from scratch.

• Experience in data processing and ingestion.

• Efficient management of files, especially large volumes of information.

• Knowledge of memory management and proper use of data structures.

• Robust management of exceptions and errors during ingestion processes.

• Ability to develop maintainable and reusable code.

• Mastery of complex SQL.

• Experience with different types of JOIN and relationships between tables.

• Aggregation functions and complex filters.

• Mastery of WHERE vs HAVING.

• Detection and handling of duplicates.

• Management of orphan records.

• Mastery of Window Functions.

• Ability to optimize queries and work with large volumes of data.

• Real experience working with large tables and data volumes.

• Creation, management, and optimization of indices.

• Interpretation of execution plans.

• Experience using EXPLAIN.

• Ability to detect bottlenecks.

• Knowledge of situations in which an index can be counterproductive.

• Experience developing ETL/ELT processes through code.

• Design of incremental and full loads.

• Error management during ingestion processes.

• Ability to design recovery and control mechanisms.

• Implementation of validations and data quality controls.

• Experience with StarRocks.

• Experience with columnar analytical databases such as ClickHouse or Apache Doris.

• Knowledge of Data Warehousing.

• Dimensional modeling:

  • Star schema

  • Fact tables

  • Dimensions

• Knowledge of OLTP vs OLAP architecture.

• Experience with Git.

• Knowledge of orchestration tools such as Apache Airflow, Cron or other Open Source solutions.

• Basic knowledge of data visualization, for example Power BI.

Data Engineer | Python + SQL + PostgreSQL

We are looking for a Data Engineer with solid experience in Python, SQL, and PostgreSQL to join a project with a strong focus on Open Source technologies.

We are looking for a hands-on profile, accustomed to developing data solutions from scratch through code. In this project, visual ETL tools or drag & drop type solutions are not used: the work is performed directly through Python scripts and advanced SQL.

The client uses StarRocks as a high-performance analytical engine, so we are looking for professionals with a solid foundation in relational databases, Data Warehousing, and SQL optimization, with the ability to adapt quickly to new technologies.

• Development of data ingestion and transformation processes using Python and SQL.

• Creation of scripts from scratch for data and file processing.

• Design and development of ETL / ELT processes completely code-oriented.

• Management of incremental loads and full loads.

• Implementation of robust error control and failure recovery mechanisms.

• Development of data quality controls through code.

• Optimization of queries and processes over large volumes of information.

• Design, creation, and optimization of indices in PostgreSQL.

• Analysis of execution plans using EXPLAIN.

• Identification and resolution of performance issues.

• Work with data structures and optimization of memory consumption during ingestion processes.

• Automation and maintenance of data processes.

• Collaboration in the design and evolution of analytical solutions.

• 100% remote.

We are not looking for profiles oriented towards traditional Big Data nor professionals whose work is based mainly on visual ETL tools.

Previous experience in Spark, Hadoop, Hive, or HDFS is not necessary. The project is mainly oriented towards relational and analytical data engineering, based on:

Python + SQL + PostgreSQL + ETL/ELT through code.

We are looking for a person who is highly technical, autonomous, and hands-on, accustomed to programming and solving problems directly through code.

It will be especially important to have a solid foundation in SQL, PostgreSQL, query optimization, and ETL process development, as well as the ability to work with large volumes of data and quickly learn new technologies such as StarRocks.