Data Engineer Must Have Technical/Functional Skills Orchestration: Apache Airflow, Apache NiFi. Programming: Java (Core), Python (for Airflow), Unix Shell Scripting. Big Data/Storage: Apache Spark, MinIO, AWS S3. Security: SSL/TLS, Certificate Management, IAM, Java Keystores. OS: Linux/Unix (RHEL/Ubuntu). Key Responsibilities: Pipeline Orchestration: Design and develop complex, reusable DAGs in Apache Airflow to automate data workflows, scheduling, and monitoring across the enterprise. Data Ingestion & Flow: Create and optimize high-volume data streams using Apache NiFi, leveraging custom processors and controller services for diverse data sources and sinks. Custom Development: Utilize Java to develop custom NiFi processors or troubleshoot the core NiFi framework, ensuring the platform meets specific architectural requirements. Object Storage Management: Implement and manage data storage solutions using MinIO and AWS S3, ensuring high availability and efficient data retrieval patterns. Large-Scale Processing: Develop and maintain Apache Spark jobs for heavy-duty data transformations, integrating them into NiFi and Airflow orchestration layers. Security & Certificate Management: Secure NiFi clusters and data flows using TLS/mTLS, managing Java KeyStores (JKS), TrustStores, and SSL certificate lifecycles to ensure data-in-motion security. System Administration: Perform environment setup, performance tuning, and troubleshooting within Unix/Linux environments, including shell scripting for task automation. Cloud Integration: Deploy and manage data infrastructure components on AWS, utilizing IAM for access control and integrating cloud-native services with hybrid data pipelines. Monitoring & Optimization: Establish robust logging and alerting for data pipelines to proactively identify bottlenecks, ensuring 99.9% reliability of data delivery. Salary Range- $100,000-$140,000 a year