PURPOSE OF THE JOB

As Senior Big Data engineer you will work on one of the world largest marketplace, dealing with one billion events per day and contribute as part of R&D self-organized team working in a challenging, innovative environment for our Customer.

MAIN TASKS AND RESPONSIBILITIES

• Work proactively on the system architecture, choosing optimal solutions to build scalable and reliable systems to process key financial data
• Data processing system optimization and tuning
• Establish monitoring system
• Ensure high quality development standards (unit/integration tests, etc.)
• Collaboration with the product management team to incorporate the needs of our customers
• Develop new generation big data pipelines, by migrating from batch processing into near-real time architecture

EDUCATION, SKILLS AND EXPERIENCE

Must have:

• 1+ years of commercial Scala development (at least)
• 5+ years of commercial JVM-based languages development experience
• Several years of experience in big data technologies and tools
• ETL or Pipeline design/implementation with Large Distributed Databases (Hive/Spark/Scala preferred)
• Hands-on experience with: Spark, Kafka, HBase, NoSQL-Databases
• Pipeline stability and performance tuning
• Working knowledge of relational databases and query authoring (SQL)
• Familiarity with the AWS-Stack (AWS EMR)
• Rigor in high code quality, automated testing, and other engineering best practices
• Experience in data validation, sampling & regression tests
• Good English (oral & written) and communication skills in general

Would be a plus:

• Commercial experience with Real-time processing systems (Spark Streaming, Storm)
• Design and development of robust distributed systems
• Having experience working with AirFlow, Hive, Presto
• Payment data processing or system design knowledge

Verifica pe BestJobs