Experiences Running Apache Flink at Very Large Scale @StephanEwen Berlin Buzzwords, 2017 1 Some large scale use cases 2 @  Various use cases • Example: Stream ingestion, route events to Kafka, ES, Hive • Example: Model user interaction sessions  Mix of stateless / moderate state / large state  Stream Processing as a Service • Launching, monitoring, scaling, updating 3 @ 4 @  Blink based on Flink  A core system in Alibaba Search • Machine learning, search, recommendations • A/B testing of search algorithms • Online feature updates to boost conversion rate  Alibaba is a major contributor to Flink  Contributing many changes back to open source 5 @ 6 @ Social network implemented using event sourcing and CQRS (Command Query Responsibility Segregation) on Kafka/Flink/Elasticsearch/Redis More: https://data-artisans.com/blog/drivetribe-cqrs-apache-flink 7 How we learned to view Flink through its users 8 System for Event–driven Applications Stateful, event-driven, event-time-aware processing Stream Processing Event-driven (streams, windows, …) Applications (event sourcing, CQRS, …) Batch Processing (data sets) 9 Event Sourcing + Memory Image periodically snapshot main memory the memory event / command event log update local variables/structures persists events (temporarily) Process 10