As a Sr. Data Engineer, the candidate will have 8+ yrs of experience and love being a problem solver, translating requirements into data metrics and schemas that are easy to understand. Strong interpersonal skills, enabling you to guide people through conversations to collect the information you need. Naturally, you are a flexible, conscientious, independent worker, capable of learning new products and technologies with the following technical skills
Good knowledge of Data structures and algorithms along with Hadoop file formats of AVRO, JSON, Parquet
Working knowledge in Python.
Experience with building Data Pipelines from the ground up - Big Data : Batch & stream data processing using AWS platform, Snowflake, Apache Spark, EMR, Hadoop, Kafka, KStream, Data as a service (API)
Proven experience with Kafka - Producer, Consumer, and Streams API.
Knowledge with KStream, KSQL, Schema registry will be a plus.
Proven with streaming and batch data processing.
Proficient in transferring unstructured into structured data.
Ability to build and maintain optimal data pipeline arc.