Apache Spark Jobs

I’m looking for an experienced consultant who can take the lead in architecting and standing up a full-scale data lakehouse, with the immediate goal of seamless data integration across my current and future sources. What I need from you: • A clear, vendor-agnostic architecture blueprint that balances performance, cost, and governance. • Hands-on implementation of the core storage layer (Delta/Parquet or comparable), compute engine (Spark or equivalent), and ingestion pipelines. • Robust security and data-quality controls baked in from day one, including role-based access, lineage, and monitoring. • Documentation and a concise knowledge-transfer session so my internal team can extend and maintain the platform confidently. Acceptance criteria: 1. A reproducible i...

Dear Data Engineers, We are currently looking for mutliple Data Engineers for one of the leading tech companies in Europe. The position is 100% remote and long-term. QUICK FACTS: - Freelancing/Contracting - Start: ASAP - 100% remote - Duration: long-term 12+ months - Capacity: full-time, part-time also possible - Language: English (no german needed!!) - German business hours REQUIREMENTS: - Experience in data engineering, ideally in Microsoft Azure environments. - Knowledge of Databricks, Apache Spark, and Medallion Architecture. - Experience with Delta Lake / Delta Tables, including concepts such as Row-Level Security. - Ability to work with Spark DataFrames, Spark SQL, and UDFs. - Good Python skills for data processing, transformation, and pipeline development. Are you interested? I...

We are looking for a freelance Subject Matter Expert (SME) who has successfully cleared the Microsoft Certified: Azure Databricks Data Engineer Associate (beta) certification to support the development of high-quality certification preparation content. Project Scope The selected SME will be responsible for: Creating practice questions aligned with the certification objectives and exam blueprint Designing scenario-based and real-world exam-style questions Providing detailed explanations and answer rationales Ensuring technical accuracy and relevance to the latest certification updates Reviewing and refining content based on feedback, if required Requirements Must have successfully cleared the Microsoft Certified: Azure Databricks Data Engineer Associate (Beta) certification Strong hands-...

I’m looking for an experienced ETL specialist (10 + years) who can step in as my on-call partner during my working hours, 8 am – 3 pm CST, all online and paid monthly. I work in a finance environment that relies heavily on Snowflake and DBT, so the focus is on keeping daily data pipelines healthy, performant, and well-tested. During each session you will help me: • Extract, transform, and load data from multiple sources. • Configure, tune, and troubleshoot Snowflake warehouses. • Design, refactor, and test DBT models. The tech stack you’ll need to be fully comfortable with includes: • SQL for complex queries and performance tuning. • Python for ad-hoc scripts, utilities, and automation. • Apache Spark for the occasional large-scale ...