Senior AWS Data & Solution Engineer

Location: New York City, NY(Hybrid)

Experience: 10+ Years

Certification Requirement: AWS Solutions Architect, AWS Data Engineer, or AWS Data Analytics Specialty Certification

Job Summary

Acunor is seeking a Senior AWS Data & Solution Engineer with strong hands-on expertise in Python, PySpark, and AWS data engineering. The ideal candidate will design and build enterprise data lakes, develop batch and event-driven data pipelines, implement secure REST APIs, and support production applications.

This role combines hands-on development, solution design, production support, and technical leadership.

Key Responsibilities

• Design, develop, and unit test Python and PySpark applications using Apache Spark.

• Build data pipelines integrating Oracle, MongoDB, DynamoDB, Amazon S3, and HDFS.

• Develop PySpark ETL pipelines using AWS Glue and Amazon EMR.

• Build Python-based AWS Lambda functions using boto3, awswrangler, pandas, JSON, and requests.

• Develop event-driven integrations using Apache Kafka, Amazon MSK, and Confluent libraries.

• Implement Apache Iceberg tables with schema evolution, CDC merges, and current-state snapshots.

• Develop REST APIs using FastAPI or Flask with Swagger/OpenAPI specifications.

• Secure API endpoints using OAuth2 and JWT authentication.

• Orchestrate workflows using AWS Step Functions and Amazon EventBridge.

• Configure AWS Glue crawlers and Data Catalog integrations.

• Optimize Spark applications, SQL queries, partitioning, compression, and file formats.

• Deploy applications using Docker, Kubernetes, and Amazon EKS.

• Build agentic AI workflows and use Copilot/GPT for development.

• Monitor and troubleshoot applications using Amazon CloudWatch and Datadog.

• Participate in code reviews, production releases, deployments, and incident resolution.

• Create technical documentation, deployment guides, and operational runbooks.

• Coordinate technical designs across stakeholders and onsite/offshore teams.

• Mentor engineers and contribute to Agile delivery and engineering standards.

Required Qualifications

• Bachelor's degree or equivalent in Computer Science or a related discipline.

• 10+ years of relevant professional experience.

• Valid AWS certification in Solutions Architecture, Data Engineering, or Data Analytics Specialty.

• Strong hands-on Python, PySpark, Apache Spark, and SQL skills.

• Experience with AWS S3, Glue ETL, Glue Data Catalog, Lambda, EventBridge, Step Functions, Athena, and EMR.

• Hands-on Kafka integration, including event publishing and consumption.

• Experience with Apache Iceberg, CDC processing, schema evolution, and snapshots.

• Experience with pandas, requests, boto3, and awswrangler.

• Experience processing Avro, Parquet, ORC, JSON, and XML data formats.

• Strong FastAPI or Flask development experience, including Swagger/OpenAPI.

• Experience implementing OAuth2/JWT authentication for REST APIs.

• Experience with Docker, Kubernetes, and Amazon EKS.

• Hands-on experience building agentic AI workflows and using Copilot/GPT for coding.

• Production troubleshooting experience using CloudWatch and Datadog.