Skip to content

aws-lambda.md

Latest commit

 

History

History
189 lines (166 loc) · 10 KB

aws-lambda.md

File metadata and controls

189 lines (166 loc) · 10 KB
Raw

AWS Lambdas playground

This part is an experiment with AWS Lambda flavors:

  • micronaut Java8 and Kotlin setups Micronaut -based lambdas, run with java8 existing AWS runtime and include lambda function plus API gateway instance. Simple, but really slow on cold starts. Here is a link with an instruction how to generate lambda from the scratch with micronaut tooling.
  • native lambdas for Java8 and Kotlin Micronaut-based lambdas with GraalVM custom runtime. It promises to solve cold start issues and keep warm timings similar to java8. Here is Micronaut build instruction.
  • serverless-nodejs setup Nodejs-based lambda. Autogenerated with Serverless framework. Used as a performance reference for load-test benchmark.

Pre-requisites

Code deployment pipeline

Serverless framework has been chosen as a cloud delivery vehicle. Please see get-started guide for setup details (few steps actually).

AWS account

Experimentation implemented with AWS platform. There is an assumption that you have AWS account. Just in case, here is AWS manual about it.

AWS role

Serverless framework needs a set of permissions in order to deploy application stack. You may find full permissions list into attached policy. This one has to be linked to AWS user. Remember to provide Serverless with AWS user configuration, the simplest approach is to export keys variables:

export AWS_ACCESS_KEY_ID=<your access key here>
export AWS_SECRET_ACCESS_KEY=<your secret key>

AWS Deployment

# call commands at the project root
# java8/kotlin/nodejs parallel build and deployment
./gradlew awsLambdaDeploy

# see the whole tasks dependency tree with this command
./gradlew awsLambdaDeploy --dry-run

# remove all components from AWS
./gradlew awsLambdaRemove

Performance validation

There are 5 configurations to test:

  • Java8 JVM
  • Java8 GraalVM native
  • Kotlin JVM
  • Kotlin GraalVM native
  • Node.js, reference one

Cold start test

AWS Lambda deployment guarantees (within our scenario) that first lambda call will experience "cold start". Therefore, the "cold start" benchmarking strategy is:

  • deploy lambdas set
  • load test each of them with 10 seconds 1-virtual user run (see run body here)
  • repeat deploy-benchmark iteration 10 times You can run the same script with:
./config/load-test/aws-deploy-cold-call.sh

Warm runs test

Lambda stay "warm" (not having init phase) some time after each call. Warm runs test uses this fact, it's scenario for each configuration:

  • trigger lambda call in case if there is going to be "cold start"
  • await 10 seconds, mainly to isolate "cold" call form main run results
  • run the same load test body 10 times Warm runs load test could be triggered by script call:
./config/load-test/aws-deploy-warm-call.sh

Please remember to modify cold and warm scripts with actual lambda URLs. You may find them with command:

# run in project root
./gradlew awsLambdaInfo

Measurements parsing

AWS-side timings captured from CloudWatch Insights. This service will help you to pull statistics with a few queries.

Load test main statistics query

fields @timestamp, @duration, @billedDuration
| filter ispresent(@duration)
| stats count(),                # calls total number
        sum(@billedDuration),   # billed time, total
        sum(@duration),         # call time sum, total
        pct(@duration, 50),     # 50%-ile for calls timings
        pct(@duration, 90),     # 90%-ile
        pct(@duration, 95),     # 95%-ile 
        pct(@duration, 99),     # 99%-ile 
        max(@duration),         # max
        min(@duration)          # min

Load test init timings query

parse @message "Init Duration: * ms" as @initduration
| filter ispresent(@initduration)
| stats count(),           # calls total number
        avg(@initduration) # avg init duration

Performance samples

Client-side measurements

These numbers captured from K6 output, while running load tests at the client-side.

min,ms 50%-ile,ms 90%-ile,ms 95%-ile,ms 99%-ile,ms max,ms calls
java8, cold 61.45 159.37 194.84 2420 6720 7630 15
java8-native, cold 57.22 61.5 183.71 193.05 317.93 807.31 114
kotlin, cold 61.47 183.67 198.71 243 0 6590 7640 15
kotlin-native, cold 56.83 62.01 168.55 210.34 312.41 823 111
nodejs, cold 56.88 61.8 147.56 191.68 246.59 406.32 116
java8, warm 55.6 60.42 143.52 202.05 402.47 493.33 123
java8-native, warm 56.33 60.27 84.49 188.46 253.71 327.64 132
kotlin, warm 56.31 60.51 153.84 191.96 326.34 376.02 122
kotlin-native, warm 56.41 61.69 142.71 165.77 233.56 276.32 129
nodejs, warm 55.89 60.97 148.85 197.83 443.78 484.94 119

AWS-side measurements

This data captured from lambdas logs and parsed with queries above.

min,ms 50%-ile,ms 90%-ile,ms 95%-ile,ms 99%-ile,ms max,ms init, avg ms calls billed,ms billed/call,ms
java8, cold 2.0 2.17 233.67 4471.92 4695.03 4739.86 2554.31 164 63300 385.97
java-native, cold 1.47 1.719 2.1784 28.9716 170.62 180.51 338.84 1125 119000 105.77
kotlin, cold 1.98 2.15 202.3 4430.62 4674.97 4889.21 2637.84 159 62100 390.56
kotlin-native, cold 1.55 1.7087 2.1784 20.1963 171.3021 181.46 331.27 1110 117400 105.76
nodejs, cold 0.84 1.03 1.23 1.35 2.78 17.11 165.22 1141 114100 100
java8, warm 1.23 1.42 1.87 2.14 19.15 290.41 0 1202 120400 100.16
java8-native, warm 1.43 1.61 1.83 1.94 21.22 178.85 0 1176 117900 100.25
kotlin, warm 1.23 1.45 1.93 2.25 17.91 217.27 0 1156 115800 100.17
kotlin-native, warm 1.45 1.63 1.88 2.01 19.03 172.9 0 1209 121200 100.25
nodejs, warm 0.79 0.97 1.17 1.27 1.59 26.41 0 1188 118800 100

Logs and stats review

# request logs and stats from Cloud Watch and see them into console
./gradlew :mn-ping-java:awsLambdaStatsGraal
./gradlew :mn-ping-java:awsLambdaStatsJvm

# same one for Kotlin lambdas
./gradlew :mn-ping-kotlin:awsLambdaStatsGraal
./gradlew :mn-ping-kotlin:awsLambdaStatsJvm
...
# shortcut calls
./gradlew :mn-ping-java:aLSG
./gradlew :mn-ping-java:aLSJ
./gradlew :mn-ping-kotlin:aLSG
./gradlew :mn-ping-kotlin:aLSJ

Some outcomes

  • JVM-based "cold" start is still the thing. It guarantees roundtrips ~5-7 seconds, which is a huge roadblock for latency-sensible scenarios. It does not matter whether lambda is packed from Kotlin or Java sources.
  • Kotlin artifact adds-up a little on top of Java8 one. Those are 38.58 MB (Java) and 44.37 MB (Kotlin).
  • GraalVM approaches Node.js in a sense of "cold" start timings and at the same time keeps the lambda door open for Java-based solutions.
  • Worth to mention that Java/Kotlin GraalVM-generated artifacts are exactly the same size, 14.62 MB.
  • Node.js demonstrated great improvements and now it is blazingly (and consistently) fast. Actually, it is the platform with the most stable performance. It's median and 99%-ile numbers are barely different. Likely it is correlated with Node.js artifact size, only 297 bytes :)