AI Data Engineer – AI Training Data
Role Overview
We are seeking a hands-on Data Pipeline Engineer to build and operate Python-based workflows that transform large document and text datasets into high-quality data for AI model training. This role combines OCR processing, data cleaning and validation, scalable pipeline development, and Azure storage management.
The ideal candidate has strong Python programming skills, experience processing datasets that exceed single-machine memory, and practical experience managing data and processing workflows in Azure.
Key Responsibilities
• Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication.
• Run and monitor large-scale batch processing jobs, investigate failures, and implement reliable retry and recovery mechanisms.
• Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies.
• Prepare training-ready datasets through filtering, text segmentation, tokenization, and conversion into agreed data formats.
• Manage bulk uploads, downloads, and synchronization between local processing environments and Azure storage; verify transfer completeness and data integrity.
• Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability.
• Optimize processing throughput, memory usage, storage consumption, and cloud costs using appropriate batch, parallel, or distributed processing approaches.
• Track dataset statistics, processing coverage, quality metrics, and exceptions; produce clear operational reports.
• Collaborate with applied scientists and model-training teams to translate data requirements into reproducible processing workflows.
• Maintain tested, documented, version-controlled code and follow established data access, security, and governance requirements.
Required Qualifications
• Demonstrated experience in data engineering, data pipeline development, or a data science role with substantial production programming responsibilities.
• Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors.
• Hands-on experience processing large structured and unstructured datasets using memory-efficient, parallel, or distributed techniques.
• Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk data transfers, storage organization, and access troubleshooting.
• Familiarity with Python data-processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet.
• Working knowledge of SQL, Git, Linux environments, and command-line tools.
• Strong troubleshooting skills, attention to data quality, and ability to communicate technical issues clearly.
Preferred Qualifications
• Experience with OCR pipelines, scanned documents, multilingual text, or document-layout processing.
• Experience with Azure Databricks, Azure Machine Learning, Azure Data Factory, or workflow orchestration tools.
• Familiarity with Hugging Face Datasets, tokenization, near-duplicate detection, and training/evaluation data separation.
• Experience with dataset lineage, reproducibility, containerized environments, and CI/CD.
Expected Deliverables
• Reliable, documented Python pipelines for OCR and training-data preparation.
• Validated, versioned datasets with metadata and quality summaries.
• Automated Azure transfer workflows with completeness and integrity checks.
• Operational monitoring and runbooks for troubleshooting, recovery, and handoff.
Explain a typical day in the role.:
• Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication.
• Run and monitor large-scale batch processing jobs, investigate failures, and implement reliable retry and recovery mechanisms.
• Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies.
• Prepare training-ready datasets through filtering, text segmentation, tokenization, and conversion into agreed data formats.
• Manage bulk uploads, downloads, and synchronization between local processing environments and Azure storage; verify transfer completeness and data integrity.
• Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability.
• Optimize processing throughput, memory usage, storage consumption, and cloud costs using appropriate batch, parallel, or distributed processing approaches.
• Track dataset statistics, processing coverage, quality metrics, and exceptions; produce clear operational reports.
• Collaborate with applied scientists and model-training teams to translate data requirements into reproducible processing workflows.
• Maintain tested, documented, version-controlled code and follow established data access, security, and governance requirements.
What is the ideal background of a candidate for this role?:
The ideal candidate has strong Python programming skills, experience processing datasets that exceed single-machine memory, and practical experience managing data and processing workflows in Azure.
What are unique selling points that would get candidates interested in your role over another?:
End-to-end AI data management and dataset curation
How will contractor performance be measured?: Data quality and risk mitigation
Where is the work able to be performed?: Onsite
If Onsite or HWA required, indicate why: Preference / collaboration / convenience
If REMOTE: Are work hours flexible to accommodate time zone variance (+/- 2-3 hours?): No
• Best vs. Average: The ideal resume would show prior company experience, hands-on data management and coding experience, and a background in data science, mathematics, computer science, or a related field.
Top 3 Must-Have HARD Skills & years of experience for each:
• Demonstrated experience in data engineering, data pipeline development, or a data science role with substantial production programming responsibilities.
• Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors.
• Hands-on experience processing large structured and unstructured datasets using memory-efficient, parallel, or distributed techniques.
**The sponsor would really like to stress that candidates must come in with hands-on experience building datasets from the ground up. They should have previously created datasets, not just worked with existing ones.**