Incedo Inc. is a high-growth Digital, Data and AI Transformation Specialist firm headquartered in New Jersey. We
\n
are a long-term strategy execution partner for Fortune 500 enterprises, operating at the intersection of business
\n
and technology across Banking & Payments, Wealth Management, Telecom, Hi-Tech, and Life Sciences.
\n
We are building Incedo 4.0 - an AI-native, execution-focused, founder-led organization designed for scale, speed,
\n
and long-term impact.
\n
Incedo delivers ROI from AI @ Scale through the “Power of 3”:
\n
• Deep domain expertise
\n
• AI & Data capabilities
\n
• Engineering & Operations excellence.
\n
\n
Job Description::
\n
We are seeking an experienced Data Engineer with strong expertise in AWS OpenSearch to design, develop, and maintain scalable data pipelines and search solutions. The ideal candidate will have hands-on experience with AWS data services, OpenSearch, Python, SQL, and cloud-native architectures to support analytics and real-time search capabilities.
\n
\n
Key Responsibilities
\n
- \n
- Design, develop, and maintain scalable data pipelines using AWS services.
- \n
- Build and optimize AWS OpenSearch clusters for indexing, search, and analytics.
- \n
- Develop ETL/ELT pipelines using Python, SQL, and AWS services such as Glue, Lambda, and EMR.
- \n
- Design and maintain OpenSearch indexes, mappings, analyzers, and query performance.
- \n
- Integrate OpenSearch with upstream and downstream applications for real-time search and analytics.
- \n
- Work with structured and semi-structured data from multiple data sources.
- \n
- Optimize OpenSearch cluster performance, shard allocation, indexing, and query execution.
- \n
- Implement Infrastructure as Code (Terraform or CloudFormation) for AWS resources.
- \n
- Build monitoring and alerting using CloudWatch, Grafana, or Prometheus.
- \n
- Collaborate with Data Scientists, Data Analysts, and application teams to support data and search requirements.
- \n
- Implement security best practices, IAM policies, encryption, and access controls.
- \n
- Troubleshoot production issues and optimize system performance.
- \n
\n
\n
Required Skills
\n
- \n
- 5+ years of experience as a Data Engineer.
- \n
- Strong experience with AWS OpenSearch (formerly Amazon Elasticsearch Service).
- \n
- Strong SQL and Python programming skills.
- \n
- Experience with AWS services:
- \n
- OpenSearch
- \n
- S3
- \n
- Glue
- \n
- Lambda
- \n
- IAM
- \n
- CloudWatch
- \n
- EC2
- \n
- EMR
- \n
- Step Functions
- \n
- Kinesis (preferred)
- \n
- Experience building ETL/ELT pipelines.
- \n
- Experience with Docker and Kubernetes (preferred).
- \n
- Knowledge of distributed search architecture, indexing strategies, and query optimization.
- \n
- Experience with Git, CI/CD, and Infrastructure as Code (Terraform or CloudFormation).
- \n
- Familiarity with Agile/Scrum methodologies.
- \n
\n
\n
Preferred Qualifications
\n
- \n
- Experience with Apache Spark or PySpark.
- \n
- Experience with Kafka or Amazon MSK.
- \n
- Experience with Snowflake, Redshift, or Databricks.
- \n
- Experience working with large-scale distributed systems.
- \n
- AWS Certification (Solutions Architect, Data Analytics, or Developer) is a plus.
- \n
- Experience with observability tools such as Grafana, Prometheus, or Splunk.
- \n
\n
\n
Nice to Have
\n
- \n
- Vector search and semantic search using OpenSearch.
- \n
- OpenSearch Dashboards/Kibana.
- \n
- Machine Learning integrations with OpenSearch.
- \n
- Experience with GenAI/RAG search architectures.
- \n
- Healthcare, Financial Services, or Retail domain experience.
- \n
\n
\n
Primary Technologies: AWS OpenSearch, Python, SQL, AWS Glue, Lambda, S3, Terraform, Docker, Kubernetes, CloudWatch, Git, CI/CD, Spark/PySpark.