Workload Simulation

Selecting a tool for workload simulation can depend on a number of things, including, but not limited to, the features it provides, the system to be tested and the reason for using it – is it a temporary deployment for ad-hoc troubleshooting, or something more long term that is part of a performance engineering strategy? In the case of a strategic framework the implementation should also align with the organisation / team’s technology preferences and way of working, as well as the environment it is to be used in (e.g. cloud, on premise). Additional considerations may include but are not limited to:

  • tests: how are they created, are they programmable, what is the engine ?
  • scenarios: what features are provided for creating and controlling workloads ?
  • protocols: does it support the application protocols in use ?
  • extensibility: can it be extended to cover new protocols or interfaces later down the line ?
  • monitoring: does it integrate with common monitoring platforms (e.g. Influxdb, Elastic) ?
  • data: what data management features are available (e.g. input formats, real-time data sharing) ?
  • automation: how well does it integrate with CI/CD tools, does it support threshold criteria ?
  • scalability: how is this best achieved, what does the tool provide for coordination ?
  • infrastructure: what type and how much is required to generate the target workloads ?
  • licensing: commercial vs open source, are any open source features paid for ?
  • management: how can results and collateral be managed ?
  • reporting: options available, can raw data be accessed, what is the precision of response times ?
  • development: how active is the community, when was it last updated ?

If feasible it can be a useful exercise to shortlist a set of tools and evaluate them against a benchmark environment to see first hand how each is used to design a test, what the quality of reporting is like, and also estimate how much capacity may be required to simulate the test workloads.

Open Source Tools

The table below outlines some of the open source tools available for workload simulation as of November 2022.

Artilleryhttps://www.artillery.ioCloud scale performance testing tool. Node.js and YAML.
AWS DLT Solution for distributed load testing. JMeter, http client.
BWMG based load testing tool, using micro EC2 instances.
Drill load testing application written in Rust.
Fortiohttps://fortio.orgGo based microservices, command line load testing tool and echo server.
Gatlinghttps://gatling.ioLoad testing framework built on Scala, Akka, Netty.
Hey Load testing tool.
JMeterhttps://jmeter.apache.orgPlugin based multi-protocol load testing tool build on Java.
K6https://k6.ioLoad testing framework, extensible to other test types.
Locusthttps://locust.ioPython based load testing tool.
nGrinder load testing framework, allows multiple concurrent tests.
Siege based url testing tool, http/https, ftp.
The Grinderhttps://grinder.sourceforge.netJython based, controller/injector model.
Tsunghttp://tsung.erlang-projects.orgErlang, multi-protocol.
Vegeta Load testing at a constant request rate.
Wrk benchmarking tool.
Summary of open source load testing tools as of November 2022.