Selecting a tool for workload simulation can depend on a number of things, including, but not limited to, the features it provides, the system to be tested and the reason for using it – is it a temporary deployment for ad-hoc troubleshooting, or something more long term that is part of a performance engineering strategy? In the case of a strategic framework the implementation should also align with the organisation / team’s technology preferences and way of working, as well as the environment it is to be used in (e.g. cloud, on premise). Additional considerations may include but are not limited to:
- Tests: how are they created, are they programmable, what is the engine ?
- Scenarios: what features are provided for creating and controlling workloads ?
- Protocols: does it support the application protocols in use ?
- Extensibility: can it be extended to cover new protocols or interfaces later down the line ?
- Monitoring: does it integrate with common monitoring platforms (e.g. InfluxDB, Elastic) ?
- Data: what data management features are available (e.g. input formats, real-time data sharing) ?
- Automation: how well does it integrate with CI/CD tools, does it support threshold criteria ?
- Scalability: how is this best achieved, what does the tool provide for coordination ?
- Infrastructure: what type and how much is required to generate the target workloads ?
- Licensing: commercial vs open source, are any open source features paid for ?
- Management: how can results and collateral be managed ?
- Reporting: options available, can raw data be accessed, what is the precision of response times ?
- Development: how active is the community, when was it last updated ?
If feasible it can be a useful exercise to shortlist a set of tools and evaluate them against a benchmark environment to see first hand how each is used to design a test, what the quality of reporting is like, and also estimate how much capacity may be required to simulate the test workloads.
Open Source Tools
The table below outlines some of the open source tools available for workload simulation as of November 2022.
| Tool | Url | Description |
|---|---|---|
| Artillery | https://www.artillery.io | Cloud scale performance testing tool. Node.js and YAML. |
| AWS DLT | https://aws.amazon.com | AWS Solution for distributed load testing. JMeter, http client. |
| BWMG | https://github.com/newsapps/bees | Python based load testing tool, using micro EC2 instances. |
| Drill | https://github.com/fcsonline/drill | HTTP load testing application written in Rust. |
| Fortio | https://fortio.org | Go based microservices, command line load testing tool and echo server. |
| Gatling | https://gatling.io | Load testing framework built on Scala, Akka, Netty. |
| Hey | https://github.com/rakyll/hey | HTTP Load testing tool. |
| JMeter | https://jmeter.apache.org | Plugin based multi-protocol load testing tool build on Java. |
| K6 | https://k6.io | Load testing framework, extensible to other test types. |
| Locust | https://locust.io | Python based load testing tool. |
| nGrinder | https://naver.github.io/ngrinder | Jython load testing framework, allows multiple concurrent tests. |
| Siege | https://github.com/JoeDog/siege | C based url testing tool, http/https, ftp. |
| The Grinder | https://grinder.sourceforge.net | Jython based, controller/injector model. |
| Tsung | http://tsung.erlang-projects.org | Erlang, multi-protocol. |
| Vegeta | https://github.com/tsenart/vegeta | HTTP Load testing at a constant request rate. |
| Wrk | https://github.com/wg/wrk | HTTP benchmarking tool. |