Data Engineer for Unstructured Data & AI Pipelines (Part‑time)

About the role

Neurons Lab is looking for a part‑time Data Engineer to build the ingestion and context layer for a European private investment group. The role focuses on unstructured data pipelines, entity resolution and AI‑ready embeddings.

Key responsibilities

• Stand up capture pipelines for calls, emails, Slack and messengers with speaker attribution and opt‑out controls.
• Backfill years of historical email, Slack, board protocols, decks and portfolio updates, ensuring correct parsing, deduplication and dating.
• Develop document parsing for PDFs, scanned board packs, spreadsheets and slide decks.
• Implement identity and entity resolution across multiple channels and data sources.
• Build chunking, embedding pipelines and load vector and graph stores as defined by the ontology.
• Implement incremental sync via connector layer, handling edits, deletions and avoiding full re‑crawls.
• Attach access scope and provenance metadata to every record at ingestion.
• Run PII detection, redaction and retention logic and provide audit evidence.
• Orchestrate pipelines with Airflow or Step Functions, adding monitoring and alerting.
• Control cost and latency through batching, tiered storage and reporting unit economics.
• Write runbooks for client hand‑over and ongoing operation.

Required profile

• 4+ years of data‑engineering experience with unstructured or semi‑structured data.
• Proven experience integrating multiple third‑party APIs and performing historical backfills.
• Experience building pipelines that feed LLM or retrieval‑augmented generation systems.
• Hands‑on experience handling sensitive personal data in regulated environments.
• Comfortable working as the sole data engineer in a small, distributed pod.

Required skills

• Python
• SQL
• Airflow or equivalent orchestration tool
• AWS and/or GCP data stack
• Vector stores (pgvector, OpenSearch, Pinecone‑class)
• Graph store technologies
• Google Workspace, Microsoft 365, Slack and CRM API integration
• Entity resolution / record linkage (deterministic and fuzzy)
• PII detection, redaction, encryption and retention
• Data lineage, provenance and audit patterns