Please take a moment to view our movie – JMeter in the Cloud, a cloud-testing platform by CloudIntelligence.
Watch this movie in HD in our Facebook pageThis document assumes you are already familiar with JMeter and you have pre-existing test-scripts you want to use. If so, you can start your load immediately without a setup fee.
If you are new to JMeter we suggest learning more about how to use JMeter.
Before launching a test in the cloud you’ll need to follow these three easy steps.
- Open an account at cloud-intelligence.com
- Purchase computing-hour credits
- Make sure your computer and browser meet the minimum requirements to run JMeter in the Cloud.
Follow this link to request an account at cloud-intelligence.com.
Bellow are additional prerequisites you will need in place before launching a test. Please visit our FAQ section to see how you can make sure that all prerequisites are in place.
Make sure Java & Javascript are enabled
In order to connect to the remote server and allow access to the remote JMeter console we use a remote desktop application by NOMACHINE. This remote desktop application uses a Java Applet. Please follow these instructions to make sure that all of these parts will operate normally during your test.
- Java installation is required for the remote connection application. You can download it from http://www.java.com/en/.
- Javascript must be enabled. Check under Internet Options => Security to make sure Javascript is enabled.
Outbound port 22 is open
We use SSH (port 22) to communicate with the remote servers, please verify that outbound port 22 is open.
Please visit our FAQ section to see how you can make sure that all prerequisites are in place.
Buy computing hours credits
After you have created an account at cloud-intelligence.com you will be required to purchase credits before running any tests. Use Buy Credits on the right hand menu to make your purchase. Once approved, these credits will appear in your account.
If you are using any of our cloud-testing services through one of our partners, you should use the Request Credits on the right hand menu to request more credits.
In order to run tests you will always need to have a positive credits balance.
You can choose to create your test-script on your local machine by downloading and using Apache JMeter or create your test-script in the cloud environment.
You can login to your account on cloud-intelligence.com, after you have created your JMeter test-script. Then use the Add Test menu function to create a new test. Follow the instructions on the page to complete the required details.
You need to choose the number of JMeterEngine(s) that will participate in the test. We recommend running up to 300 threads/virtual users per one JMeterEngine if you are using HTTP/S protocol. If you are using any other protocols such as Web Services we recommend up to 70 threads/virtual users. The number of supported threads depends on your script intensity and available resources. The more intense your script – the less threads a single JMeterEngine can support.
You can optionally fill the command line arguments. For example, if you upload a file named: my_config.jmx, the command line argument should include: -t my_config.jmx. Please make sure this file is uploaded to your account using the Files tab (see instructions below).
You can choose the specific geographic location from which you wish to run your load. This selection can be modified before each run, that way you can run the same test several times and each time choose a different location in the world. We provide four global locations, two in the US, and one in Europe and one in Asia.
Don't forget to save your test.
Operations on Files
Use the Files tab to upload your test-script (a file that ends with JMX extension). Upload any other files you wish to use during the load test.
The Files tab will appear only after you save the test for the first time.
At this point:
- The Files tab is accessible at all times. You can use it to upload, download and delete files.
- All files in your account are downloaded to the remote JMeter servers at the beginning of each test.
- During the test, files in your account are synchronized every few minutes.
- You cannot delete files during the load test. Files can be deleted only once the test has ended or before it has started.
- Prior to terminating your test session please make sure all files are synchronized.
Launch Servers
Once the test is correctly configured, you can launch the servers and run the load.
In the Add/Edit Test page, you can change the advanced default configuration in “Advanced Test Properties” section to fit your specic needs.
Extended JMeterEngine Resources Configuration
The JMeter in the Cloud default configuration includes running a single JMeter console on a dedicated server coupled with additional servers running two JMeterEngine(s) each. As each server is equipped with a dual core CPU and 1.7GB of memory, we limit the memory available for the console to 1GB and for each JMeterEngine to 512MB. In most cases this configuration should be more than enough. That being said, some scripts can require more resources than others.
For the purpose of more intensive scripts we provide an extended resources configuration. With this configuration each JMeterEngine runs on a dedicated server. The memory available for the JMeterEngine is limited to 1.5GB, triple the amount provided by the default configuration. Because this configuration uses one server for each JMeterEngine it requires additional computing hours when compared with the default configuration.
JMeter Version
While you can choose to use JMeter version 2.3.4, please note that by default JMeter version 2.3.2 is selected. You should use the JMeter version you are using on your local computer to create your test scripts.
Slow Internet Connection
Enable this check box only if you are using a low bandwidth Internet connection. In any case we highly recommend using JMeter in the Cloud with a strong Internet connection only. We recommend not using JMeter in the Cloud on a virtual desktop (for example using CITRIX). We are providing a remote application access to the remote JMeter servers. This requires a strong bandwidth Internet connection.
Dedicated IPs
In some situations, such as a when you are connecting from behind a corporate Firewall, you would want the JMeterEngines to have dedicated IPs. We can assign dedicated IPs to the JMeterEngines. Dedicated IPs are an expensive resource. If you wish to have dedicated IPs assigned to you, please contact us.
JMeter user & system properties
In case you need to add properties to JMeter’s default “user.properties” file, go to the Files tab of the relevant test and upload your “user.properties” file. JMeter in the Cloud will identify it and will add it to the JMeterEngines and console servers. Similarly, "system.properties" can be uploaded to update system properties.
New jar files
Users that develop and/or enhance their JMeter components can upload their own JMeter components and add-ons.
To do this, jar your JMeter components and then upload the jar files. JMeter in the Cloud will identify the uploaded jar files and will automatically copy them to the JMETER_HOME/lib/ext directiry.
System hosts file
You can control the “hosts” file of the launched servers. As a result, you are able to add additional local DNS entries as if you were editing the hosts file locally. One important use for this feature is to provide permanent DNS resolution for certain DNS names. This can reduce the number of times JMeter in the Cloud calls on your DNS for name resolution.
To do this, upload your “hosts.txt” file. This file will replace the "/etc/hosts" file on every launched server. Please note that the new hosts file should include at least one line of the form: 127.0.0.1 localhost. The name of the file must be "hosts.txt".
To launch all servers related to a test, use the Launch Servers menu function on the right hand side. Clicking the Launch Servers link will launch a reservation of servers in the cloud. The status of this operation appears in the status area on the middle of the page. Launching a reservation can take up to three minutes. If you are using dedicated IPs, it can take up to five minutes.
The servers are launched and their status will be automatically updated. Wait to be notified that all servers are ready to run.
Once all servers are launched an applet will appear allowing access to the remote JMeter console through the NOMACHINE remote connection application.
In case Java is not installed on your local machine, you will be asked to install it. Click here to download Java and follow the Java installation wizard
The NOMACHINE confirmation pop-up will be displayed. NOMACHINE is the remote connection application through which JMeter console is displayed on your local machine. Please check the “Always trust content from this publisher” option and click on the Run button.
If this is the first time you launch a test, the NOMACHINE applet will install a small component on your PC that will enable the remote session. This process can take up to two minutes for the first time.
Click the Continue button in order to start the remote connection session.
Wait for the wizard to finish the setup process.
Click on Yes in order to approve the connection with the remote servers in the cloud.
Wait for the JMeter log and console windows to open, your previously uploaded test-script is available for you to run as if you were working from your local machine.
Once servers are launched and the applet appears, click on Continue button in order to start the remote connection session.
You can see the applet establishing the remote connection. Please wait for this process to complete.
Click on Yes in order to approve the connection with the remote servers in the cloud.
Once the JMeter console window is open, your previously uploaded test-script is available for you to run. Please note that you can change the script and save the changes through the remote JMeter console, the changes also affect the JMeter file you can download from the test page.
You can choose to run your load on all servers simultaneously or start each JMeterEngine separately. We recommend running no more than 300 concurrent threads for each JMeterEngine.
In order to run all servers simultaneously, click on the "Remote Start All" option under the “Run” menu:
In order to start each JMeterEngine separately, select "Remote Start" option under the “Run” menu. All available JMeterEngines are listed; click on the one you want to start:
You can also run the load on the JMeter console by clicking on the “Start” option under the “Run” menu. This is the same as running another JMeterEngine.
Please note that we do not recommend running a load on the JMeter Console. This is because it is already consuming massive amounts of CPU capacity for calculations related to aggregating remote JMeterEngines results. Doing so may cause the JMeter Console to freeze.
The same can be done for stopping the load (by choosing “Remote Stop All”, selecting a JMeterEngine from "Remote Stop" list or clicking on “Stop” option).
JTL files are an inherent functionality of JMeter. You can read more about JTL files in the JMeter user guide. With this functionality you are able to have real-time reports including detailed graphs of tests results.
In order to view real-time reports, all you need to do is specify a JTL file name in one of JMeter's listeners. For example, add "Aggregate Report" listener and specify a JTL file name. Make sure the file ends with the extension: "jtl". Also, please make sure that the JTL file is saved as XML and not CSV. Once the test is running, you can view/download the JTL file itself and the resulting reports using the JTLs and/or Files tabs.
All graphs are displayed in the JTLs tab, click on the graph’s thumbnail in order to see it. Refresh the page in order to see the updated graph.
You can press on the graph to download the image and include it in your report. Clicking on the JTLs tab generates all available reports. Make sure you press this tab at least once before terminating your test session in order for the graphs to be generated. Please note that these reports will also be available for download after terminating the reservation in the Files tab, under the Zipped folder.
Available Graphs
To see an example of a live reports screen, please visit our demo section.
There are two types of reports:
- Aggregated Reports
- Detailed Reports
Aggregated Reports
Aggregated reports present various distributions of reponse times, latencies and errors throughout the entire test. These reports can draw the entire picture in a single snapshot.
Aggregated Reports include:
- Two graphs: Response time distribution by percentage from the entire test and according to the number of samples
- Two graphs: Latency distribution by percentage from the entire test and according to the number of samples
- Two graphs: Error/Response Code distribution by percentage from the entire test and according to the number of samples
|
|
|
Detailed Reports
Detailed reports present the average value of each measure per minute throughout the entire test. Measures include:
- Response time
- Latency
- Hits per minute
- Number of concurrent users (Active Threads)
- Errors
There is one view for all URLs that were included in the test and detailed views for each unique URL.
Detailed reports include:
- Response Time and Latency Vs Users (Generating this graph requires a user action. Please read our FAQ section for further details)
- Response Time and Latency Vs Hits
- Response Time and Latency Vs Errors
- Hits Vs Users (Generating this graph requires a user action. Please read our FAQ section for further details)
- Hits Vs Errors
To see an example of a live reports screen, please visit our demo section.
Most of these reports are available only through JMeter in the Cloud and not through the regular JMeter interface.
Although JMeter does support few graphic reports through the regular interface it is not recommended to enable them as they consume a lot of memory and will result in the console freezing under heavy load. Use the above mentioned reports instead. They have no affect on JMeter during the test run and are accessible after the test had ended.
Generating "Users vs Load Time Average" report requires the following specific actions:
- Using the JMeter Console, add a "Config Element -> User Defined Variables" .
- Add a variable named: "in". This variable's value should be the following: "${__P(InstanceID)}".
These actions allow the identification of each individual JMeterEngine. If these actions are not performed the graph will show the maximum users per one JMeterEngine. For example, assuming there are 4 JMeterEngines, each running 50 users. Without these specific actions, the maximum number of users will show 50. With these actions performed, the maximum number of users will reach 200.
Log Files
CloudIntelligence enables you to monitor each JMeterEngine log and each server CPU and Network bandwidth performance logs. These are available in the Logs tab. Click on the requested tab name to display the relevant information. The logs are automatically refreshed every 30 seconds.
Through the Logs tab you can view the last added lines of each log file. You can further download or view the entire file. Large log files (over 10 MB) are not displayed. You can only download a large log file.
We highly recommend to monitor the logs at the beginning and during the test. Any problem or error will appear in the log files.
Network Information
In addition, you can view all public DNS names involved in the test session in the Network Information tab. This can be used to configure your firewall to allow access from the specified domain names. Please note that each test session uses different domain names:
Availability
Please note that these files will also be available for download after terminating the reservation in the Files page, under the Zipped folder.
If you wish to save and download the table load results displayed in JMeter GUI, click on the Save Table Data button at the bottom of the results view. In the pop-up that opens, save the results in a csv file under the default JMeter folder. The file name must NOT contain spaces.
The results file is now saved in cloudIntelligence website, in your test page Files tab where it is available for download
The file is also available for download after terminating the test, from the Files tab.
The "Terminate all but Console" is a special option that allows you to manage reports and results without having to pay for the servers' time other than on the Console.
By using this option you will terminate all of the load servers, but keep the GUI console open. In this case all load servers are terminated, but the load results, logs and graphs are still available for viewing and downloading from the CloudIntelligence website.
You are only billed according to consumed computing hours. Once you terminate all load servers but the Console's, you are billed only for the Console time.
To terminate only the load servers, click on Terminate All But Console button on the right-hand menu:
Once you finish the test you can download JTL files, graphs and CSV files to your local machine to prepare your report. Use the Files tab to download any files generated by JMeter and saved during the run. Use the JTLs tab to download any JTL or graphs you wish to use in your report.
Do not forget to use the Terminate Test button in order to terminate the remaining server.
Once you are done with your load tests, click Terminate Test on the right-hand menu and then click OK on the confirmation pop-up, in order to free the allocated servers and confirm the termination.
From your Load Testing Home page you can verify that the load test has terminated and is no longer active.
Another option is to first terminate all of the load servers, but keep the GUI console open. In this case the load is still terminated, but the load results are available for viewing and uploading to the website. To terminate only the load servers, click on the Terminate All But Console button on the right-hand menu:
When you no longer need to view the JMeter GUI console, click on Terminate Test button in order to terminate the remaining server.
All test runs logs, jmx scripts, saved results and graphs files are saved and available for download during and after the load run from the Files page.
Logs, JMeter jtl files and graphs images are zipped into one file located under the Zipped folder.
JMeter has some limitations especially when it is run in a distributed environment. Following these guidelines will assist in creating a real and continuous load:
- Do not use the Console to create your load. You can use the console for debugging purposes or to run a small load from it to make sure the script is running correctly. The GUI consumes a lot of memory under heavy load, therefore the console server by itself can not sustain a heavy load.
- Use the remote servers to create the load. Use the "Remote Start All" or "Remote Start" individual servers. Further information can be found in our User Guide section.
- Limit the number of threads per engine to 250. This means the total number of threads generated by your test plan should be less than 250. The overall number of threads running from JMeter in the Cloud will be the total number of threads multiplied by the number of JMeterEngines. For example, a test plan with 200 threads and 4 JMeterEngines, will generate a load of 800 threads.
- Disable the "View Result Tree" listener as it consumes a lot of memory and can result in the console freezing or JMeter running out of memory. It is, however, safe to use the "View Result Tree" listener with only "Errors" checked.
- Disable all JMeter graphs as they consume a lot of memory. You can view all of the real time graphs using the JTLs tab in your web interface. Further information can be found in our User Guide section.
- Monitor the logs. Any errors in the test plan or the test itself will appear in the log files that are available online under the Logs tab. For example: Out of Memory, disconnections etc.
- Do not forget to erase the local path from CSV Data Set Config if used. Further information can be found in our FAQ section.
Best Practice Using JTL files
If you are using JTL files, we recommend following these guidelines:
- Assign a meaningful JTL file name. For example: 150210-4000t-1.jtl. Do not use spaces in the JTL file name.
- Make sure that the JTL file is saved in XML format and not CSV.
- Visit the JTLs tab at least once to generate the real-time reports.
- Use the "Terminate all but Console" feature to finish your reports and download all graphics and CSV files.
- Enable the Users vs Load Time Average report by following these instructions.
- Use short and meaningful names for labels identifying controllers and samplers (for example HTTP requests). If you use long names, the names themselves will take the better part of the graphs generated using the JTLs tab.