ECS & Docker: Secure Async Brennan Saeta

Transcription

1 ECS & Docker: Secure Async Brennan Saeta

2

3 The Beginnings million learners worldwide 4 partners 10 courses

4 Education at Scale 18 million learners worldwide 140 partners 1,800 courses

5 Outline Evolution of Coursera s nearline execution systems Next-generation execution framework: Iguazú Iguazú application deep dive: GrID evaluating programming assignments

6 Key Takeaways What is nearline execution, and why it is useful Best practices for running containers in production in the cloud Hardening techniques for securely operating container infrastructure at scale

7 A history of nearline execution

8

9 Coursera Architecture (2012) PHP Monolith

10 Early days - Requirements Video re-encoding for distribution Grade computation for 100,000+ learners Pedagogical data exports for courses

11 Coursera Architecture (2012) PHP Monolith

12 Cascade Architecture Cascade PHP Monolith PHP Monolith

13 Cascade Architecture PHP Monolith Queue Cascade PHP Monolith

14 Upgrading to Scala Re-architecting delayed execution for our 2 nd generation learning platform.

15 Upgrading to the JVM Leverage mature Scala & JVM ecosystems for code sharing JVM much more reliable (no memory leaks) New job model: scheduled recurring jobs. Named: Saturn

16 Saturn Architecture Online Serving Scala/micro-service architecture Service A Service B Service C C* C*

17 Saturn Architecture Online Serving Scala/micro-service architecture Service A Saturn Service B Service C C* C*

18 Saturn Architecture Service B Service A Service C Saturn ZK Ensemble C* C*

19 Saturn Architecture Service B Service A Service C Saturn Leader ZK Ensemble C* C*

20 Problems with Saturn Single master meant naïve implementation ran all jobs in same JVM Huge CPU top of the hour OOM Exceptions & GC issues

21 Enter: Docker Containers allow for resource isolation! CC-by-2.0

22 Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code Resource Isolation Clusters / HA Great developer workflow Scheduled Jobs

23 Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code Resource Isolation Clusters / HA Great developer workflow Scheduled Jobs

24 Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code Resource Isolation Clusters / HA Great developer workflow Scheduled Jobs

25 Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code Resource Isolation Clusters / HA Great developer workflow Scheduled Jobs

26 Supported Features Platform Saturn Docker Amazon ECS??? Run code Resource Isolation Clusters / HA Great developer workflow Scheduled Jobs

27 Solution: Iguazú Marissa Strniste ( CC-BY-2.0

28 Solution: Iguazú Framework & service for asynchronous execution Optimized Scala developer experience for Coursera Unified scheduler supports: Immediate execution (nearline) Scheduled recurring execution (cron-like) Deferred execution (run time X) Marissa Strniste ( CC-BY-2.0

29 Iguazú Architecture ECS API Devs Iguazú Admin Iguazú Scheduler SQS Iguazú Backend Iguazú Frontend Iguazú Workers Services Services Cassandra Users

30 Iguazú Architecture ECS API Devs Iguazú Admin Iguazú Scheduler SQS Queue Iguazú Backend Iguazú Frontend Iguazú Workers Services Services Cassandra Users

31 Iguazú Architecture ECS API Devs Iguazú Admin Iguazú Scheduler SQS Queue Iguazú Backend Iguazú Frontend Iguazú Workers Services Services Cassandra Users

32 Iguazú Architecture ZK Ensemble ECS API Devs Iguazú Admin Iguazú Scheduler SQS Queue Iguazú Backend Iguazú Frontend Iguazú Workers Services Services Cassandra Users

33 Iguazú Architecture ZK Ensemble ECS API Devs Iguazú Admin Iguazú Scheduler SQS Queue Iguazú Backend Iguazú Frontend Iguazú Workers Services Services Cassandra Users

34 Autoscale, autoscale, autoscale!

35 Autoscaling Iguazú ECS Shutdown Lifecycle Notification Poll Worker Job Status Autoscaling Proceed Iguazu All finished ECS API Terminate EC2 Worker EC2 Worker EC2 Worker

36 Failure in Nearline Systems Most jobs are non-idempotent Iguazú: At most once execution Time-bounded delay Future: At least once execution With caveats

37 Iguazú adoption by the numbers ~100 jobs in production >100 different job schedules >1000 runs per day

38 Iguazú Applications Nearline Jobs Pedagogical Instructor Data Exports System Integrations Course Migrations Scheduled Recurring Jobs Course Reminders System Integrations Payment reconciliation Course translations Housekeeping Build artifact archival A/B Experiments

39 While containers may help you on your journey, they are not themselves a destination. CC-by-2.0

40 Writing an Iguazu Job class (abclient: AbClient, API) extends AbstractJob { override val reservedcpu = 1024 // 1 CPU core override val reservedmemory = 1024 // 1 GB RAM } def run(parameters: JsValue) = { val experiments = abclient.findforgotten() logger.info(s"found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendreminder(experiment.owners, experiment.description) } }

41 Writing an Iguazu Job class (abclient: AbClient, API) extends AbstractJob { override val reservedcpu = 1024 // 1 CPU core override val reservedmemory = 1024 // 1 GB RAM } def run(parameters: JsValue) = { val experiments = abclient.findforgotten() logger.info(s"found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendreminder(experiment.owners, experiment.description) } }

42 Writing an Iguazu Job class (abclient: AbClient, API) extends AbstractJob { override val reservedcpu = 1024 // 1 CPU core override val reservedmemory = 1024 // 1 GB RAM } def run(parameters: JsValue) = { val experiments = abclient.findforgotten() logger.info(s"found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendreminder(experiment.owners, experiment.description) } }

43 Writing an Iguazu Job class (abclient: AbClient, API) extends AbstractJob { override val reservedcpu = 1024 // 1 CPU core override val reservedmemory = 1024 // 1 GB RAM } def run(parameters: JsValue) = { val experiments = abclient.findforgotten() logger.info(s"found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendreminder(experiment.owners, experiment.description) } }

44 Writing an Iguazu Job class (abclient: AbClient, API) extends AbstractJob { override val reservedcpu = 1024 // 1 CPU core override val reservedmemory = 1024 // 1 GB RAM } def run(parameters: JsValue) = { val experiments = abclient.findforgotten() logger.info(s"found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendreminder(experiment.owners, experiment.description) } }

45 Testing an Iguazu job

46 The Hollywood Principle applies to distributed systems. CC-by-2.0

47 Deploying a new Iguazu Job Developer merge into master done Jenkins Build Steps Compile & package job JAR Prepare Docker image Pushes image into registry Register updated job with Amazon ECS API

48 Invoking an Iguazú Job // invoking a job with one function call // from another service via REST framework RPC val invocationid = iguazujobinvocationclient.create(iguazujobinvocationrequest( jobname = "exportquizgrades", parameters = quizparams))

49 A clean environment increases reliability. CC-by-2.0

50 Evaluating Programming Assignments An application of Iguazú

51

52

53 Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure

54 Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure

55 Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure

56 Solution: GrID Service + framework for grading programming assignments Builds on Iguazú Named for Tron s digital frontier Backronym: Grading Inside Docker Patrick Hoesly ( CC-BY-2.0

57 High-level GrID Architecture GrID S3 Bucket Learners VPC Firewalls Grading Machines Iguazú ECS APIs Coursera Production Account Coursera GrID Grading Account

58 High-level GrID Architecture GrID S3 Bucket Learners VPC Firewalls Grading Machines Iguazú ECS APIs Coursera Production Account Coursera GrID Grading Account

59 High-level GrID Architecture GrID S3 Bucket Learners VPC Firewalls Grading Machines Iguazú ECS API Production Acct GrID Grading Account

60 High-level GrID Architecture GrID S3 Bucket Learners VPC Firewalls Grading Machines Iguazú ECS API Production Acct GrID Grading Account

61 Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure

62 Programming Assignments

63 The Security Challenge Compiling and running untrusted, arbitrary code on our cluster in near real time. Would you like to compile and run C code from random people on the Internet on your servers?

64 FROM redis FROM ubuntu:latest FROM jane s-image

65 Security Assumptions Run arbitrary binaries Instructor grading scripts may have vulnerabilities Grading code is untrusted Unknown vulnerabilities in Docker and Linux name-spacing and/or container implementation

66 Security Goals Prevent submitted code from: impacting the evaluation of other submissions. disrupting the grading environment (e.g., DoS) affecting the rest of the Coursera learning platform

67 Grading assignment submissions CC-by-2.0

68

69 Alice s Submission Alice s Container Grader Bob s Submission Bob s Container Grader Mallory s Submission Mallory s Container Grader Kernel RAM CPU CPU CPU CPU Disk

70 Alice s Submission Alice s Container Grader Bob s Submission Bob s Container Grader Mallory s Submission Mallory s Container Grader Kernel RAM CPU CPU CPU CPU Disk

71 Alice s Submission Alice s Container Grader Bob s Submission Bob s Container Grader Mallory s Submission Mallory s Container Grader Kernel RAM cgroups CPU cgroups CPU cgroups Disk

72 Alice s Submission Alice s Container Grader Bob s Submission Bob s Container Grader Mallory s Submission Mallory s Container Grader Kernel RAM cgroups CPU cgroups CPU cgroups Disk

73 Alice s Submission Alice s Container Grader Bob s Submission Bob s Container Grader Mallory s Submission Mallory s Container Grader Kernel RAM cgroups CPU cgroups CPU cgroups Disk blkio limits & btrfs quotas

74 Alice s Submission Alice s Container Grader Bob s Submission Bob s Container Grader Mallory s Submission Mallory s Container Grader Kernel RAM cgroups CPU cgroups CPU cgroups Disk blkio limits & btrfs quotas

75 Attacks: Kernel Resource Exhaustion Open file limits per container (nofile) nproc Process limits Limit kernel memory per cgroup Limit execution time

76

77 Alice s Submission Alice s Container Grader Bob s Submission Bob s Container Grader Mallory s Submission Mallory s Container Grader Kernel cgroups, ulimits RAM cgroups CPU cgroups CPU cgroups Disk blkio limits & btrfs quotas Network

78 Attacks: Network attacks Attacks: Bitcoin mining DoS attacks on other systems Access Amazon S3 and other AWS APIs Defense: Deny network access

79 Docker Network Modes NetworkDisabled too restrictive Some graders require local loopback Feature also deprecated --net=none + deny net_admin + audit network Isolation via Docker creating an independent network stack for each container github.com/coursera/amazon-ecs-agent

80 CC-by-2.0

81 CC-by-2.0

82 CC-by-2.0

83 Defense in Depth Mandatory Access Control (App Armor) Allows auditing or denying access to a variety of subsystems Drop capabilities from bounding set No need for NET_BIND_SERVICE, CAP_FOWNER, MKNOD Deny root within container

84 Deny Root Escalations We modify instructor grader images before allowing them to be run Clears setuid Inserts C wrapper to drop privileges from root and redirect stdin/stdout/stderr Run cleaning job on another Iguazú cluster Run Docker in Docker! Docker 1.10 adds User Namespaces

85 If all else fails Utilizes VPC security measures to further restrict network access No public internet access Security group to restrict inbound/outbound access Network flow logs for auditing Separate AWS account Run in an Auto Scaling group Regularly terminate all grading EC2 instances

86 Other Security Measures Utilize AWS CloudTrail for audit logs Third-party security monitoring (Threat Stack) No one should log in, so any TTY is an alert Penetration testing by third-party red team (Synack)

87 Lessons Learned - GrID Building a platform for code execution is hard! Carefully monitor disk usage Run the latest kernels Latest security patches btrfs wedging on older kernels Default Ubuntu kernel not new enough!

88 Reliable deploy tooling pays for itself.

89 Thank you! Brennan Saeta GrID lead Frank Chen Iguazú Lead

90 Questions? Brennan Saeta GrID lead Frank Chen Iguazú Lead