Develop proof-of-concept prototype with fast iteration and experimentation.• Develop and maintain design documentation, test cases, performance and monitoring and performance evaluation using Git, Crontab, Putty, Jenkins, Maven, Confluence, ETL, Automic, Zookeeper, Cluster Manager• Perform continuous integration and deployment using Jenkins and Git Design data integration pipeline architecture and ensure successful creation of the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Spark, SQL, HQL and other technologies.• Build robust and scalable applications using SQL, Scala/Python and Spark.•