JMeter in the Cloud - FAQ

If you expect your JTL file to contain data, however it is empty then you probably have a problem with your test script. Saving the data to a JTL file depends on the script and how JMeter interprets it.

If the JTL file was created but is empty, one can only assume that JMeter decided according to some criteria not to write any data to the file. For example, if the Errors check-box was enabled and there were no errors.

It is hard to identify the problem once the test is terminated, however, it is recommended to follow these guidelines to avoid the situation altogether:

1. View the JTL file halfway through the test to validate that the data is being recorded.
2. Generate reports halfway through the test to validate that reports can generate out of the data.
3. Always save the table data. It is recommended to save the table data just before terminating the test. Read more in our User Guide section.

If the JTL file is empty, it usually means there is a problem with the script. The best thing is to identify the problem. If you cannot identify the problem, you can bypass the problem by adding another listener using another JTL file.

If the problem persists, use our support to identify the problem in real time.

Range of IP addresses for the JMeterEngine(s). In case one needs to "whitelist" them in the corporate firewall.

US East (Northern Virginia)

216.182.224.0/20 (216.182.224.0 - 216.182.239.255)

72.44.32.0/19 (72.44.32.0 - 72.44.63.255)

67.202.0.0/18 (67.202.0.0 - 67.202.63.255)

75.101.128.0/17 (75.101.128.0 - 75.101.255.255)

174.129.0.0/16 (174.129.0.0 - 174.129.255.255)

204.236.192.0/18 (204.236.192.0 - 204.236.255.255)

184.73.0.0/16 (184.73.0.0 – 184.73.255.255

US West (Northern California)

204.236.128.0/18 (216.236.128.0 - 216.236.191.255)

184.72.0.0/18 (184.72.0.0 – 184.72.63.255)

EU (Ireland)

79.125.0.0/17 (79.125.0.0 - 79.125.127.255

Asia Pacific (Singapore)

175.41.128.0/18 (175.41.128.0 - 175.41.191.255)

JMeter in the Cloud leverages numerous open-source technologies,  Cloud-Computing resources and proprietary software in order to present the best solution at an affordable price for your Load-Testing requirements.

It's configured, It's installed, It's cost-effective

With JMeter in the Cloud you can save a lot of time and money by using a pre-configured distributed JMeter environment already deployed on strong servers.

There is no need to purchase or lease any hardware. There is no need to install or configure any software. There is no setup. Within minutes you can generate a load of thousands of concurrent virtual users running tens of servers from numerous global locations.

You can use JMeter in the Cloud on-demand and pay according to consumption. You can use it for only one hour or for one hundred hours and pay according to the number of consumed Computing Hours. There is no commitment and no upfront fee.

JMeter in the Cloud allows you to save money by terminating any servers that are no longer required. Once you complete the test and wish to prepare your report, you can easily terminate all the JMeterEngines and stop paying for their Computing Hours. You can continue to work on the Console alone and pay only for its Computing Hours. For further information about the "Terminate all but Console" option please visit our FAQ section.

Realtime Reports

JMeter in the Cloud presents numerous realtime graphical reports in addition to what is already available using Apache JMeter. For further information about available reports please visit our FAQ section or visit our live Realtime Reports demo page.

Realtime Monitoring of JMeterEngines

You are able to monitor the JMeterEngine servers in real time and identify any problems causing the servers to suffocate or perform in a way that is not satisfactory.

Realtime Monitoring of Target Servers

With CloudMonitoring you are able to monitor your target servers in real time. CloudMonitoring is able to monitor many operating systems (Windows, Unix, Linux), Internet Servers (IIS, Apache), data bases (MySQL, MSSQL) and more.

Numerous Geographical Locations

You are able to launch your test from numerous global locations (US, EU and Asia) simply by choosing the test origin.

Central Repository

You can manage numerous tests with different configurations. All logs and reports are saved per each run and are accessible at all times.

Accessible Logs

All logs of all JMeterEngines and Console are accessible as part of the JMeter in the Cloud interface and are updated in real time. This feature is very important as any problem occurring during the load-run will definitely appear in the log files. For further information about working with logs please visit our User Guide section.

JMeter in the Cloud can generate reports in real time based on JMeter's inherent JTL functionality. The reports are presented in the JTL tab during the session run time.

To see an example of a live reports screen, please visit our demo section.

There are two types of reports:

1. Aggregated Reports
2. Detailed Reports

Aggregated Reports

Aggregated reports present various distributions of response times, latencies and errors throughout the entire test. These reports can draw the entire picture in a single snapshot.

Aggregated Reports include:

1. Two graphs: Response time distribution by percentage from the entire test and  according to the number of samples
2. Two graphs: Latency distribution by percentage from the entire test and  according to the number of samples
3. Two graphs: Error/Response Code distribution by percentage from the entire test and  according to the number of samples
4. One graph: Throughput distribution

Detailed Reports

Detailed reports present the average value of each measure per minute throughout the entire test.

Measures include:

1. Response time
2. Latency
3. Hits per minute
4. Number of concurrent users (Active Threads)
5. Errors
6. Throughput (in Kilobytes)

There is one view for all URLs that were included in the test and detailed views for each unique URL.

Detailed reports include:

1. Response Time and Latency Vs Users (Generating this graph requires a user action. Please read our FAQ section for further details)
2. Response Time and Latency Vs Hits
3. Response Time and Latency Vs Errors
4. Response Time and Latency Vs Throughput
5. Hits Vs Users (Generating this graph requires a user action. Please read our FAQ section for further details)
6. Hits Vs Errors
7. Hits Vs Throughput

To see an example of a live reports screen, please visit our demo section.

If you wish to see a different type of report, contact us and let us know.

JMeter is used by a large community. You can easily learn to use JMeter from resources found on the web. You can use forums to ask questions ang get responses very fast.

Below you can find few websites offering JMeter tutorials and discussion forums related to JMeter. We will make sure this list will be refreshed and updated with new websites found by our team.

We also offer personal JMeter training using few online sessions. If you wish to receive personal JMeter training please contact us.

JMeter Tutorials

http://jakarta.apache.org/jmeter/usermanual/index.html
http://www.scribd.com/doc/7499267/Load-Testing-With-JMeter
http://www.javapassion.com/handsonlabs/javatestjmeter/
http://www.youtube.com/watch?v=KI6u5pclYIw
http://www.roseindia.net/jmeter/using-jmeter.shtml
http://zacster.blogspot.com/2008/03/quick-howto-to-setup-jmeter.html
http://www.vimeo.com/4025715

JMeter Forums

http://www.sqaforums.com/postlist.php?Cat=0&Board=UBB54
http://old.nabble.com/JMeter-f296.html

We also offer personal JMeter training using few online sessions. If you wish to receive personal JMeter training please contact us.

It is very easy to use "CSV Data Set Config" in the cloud. Upload your CSV files using the Files tab.

In the "Filename:" field in the "CSV Data Set Config" use only the file name without any path. If you are using existing script you have created in your local machine, do not forget to erase the local path.

For further information about how to use "CSV Data Set Config" with JMeter follow this link.

If you want to use multiple JMeterEngines with a different CSV file per JMeterEngine, you will need to upload a different file for each JMeterEngines and use a parameter to differentiate each file.

Each JMeterEngine uses a distinct parameter ${__P(InstanceID)}. Please note that the "InstanceID" values start from 1 for the JMeterEngines. The value of "InstanceID" for the JMeter console is 0.

In the example above, incase you are using two JMeterEngine(s), you can upload files named file1.csv and file2.csv. In the "Filename" field you should specify: file${__P(InstanceID)}.csv.

The first JMeter engine will read file1.csv. The second will read file2.csv.

If this information is not sufficient, please contact our support.

Generating "Users vs Load Time Average" report requires the following specific actions:

1. Using the JMeter Console, add a "Config Element -> User Defined Variables" .

1. Add a variable named: "in". This variable's value should be the following: "${__P(InstanceID)}".

These actions allow the identification of each individual JMeterEngine. If these actions are not performed the graph will show the maximum users per one JMeterEngine. For example, assuming there are 4 JMeterEngines, each running 50 users. Without these specific actions, the maximum number of users will show 50. With these actions performed, the maximum number of users will reach 200.

Under Advanced Test Properties, enable the Extended JMeterEngine resources configuration option.

JMeter in the Cloud default configuration includes running a single JMeter console on a dedicated server and additional servers running two JMeterEngine(s) on each. As each server is equipped with a dual core CPU and 1.7GB of memory, we limit the memory available for the console to 1GB and for each JMeterEngine to 512MB. In most cases this configuration should be more than enough. That said, JMeter will treat scripts differently according to their intensity and memory consumption.

For example let's consider two scripts:

Intense: 200 threads, doing 100 samples per thread with almost no delay between each sample.

Moderate: 300 threads, doing 10 samples per threads with few seconds delay between each sample.

JMeter will find the Intense script harder to execute compared to the Moderate script. The reason is that the Intense script requires more resources in terms of CPU power and memory than the Moderate one.

JMeter can suffocate under intensive scripts. One can easily see if his or hers script is too intensive for the available resources. Memory problems will appear in the logs.

For the purpose of more intensive scripts we provide an extended resources configuration. With this configuration each JMeterEngine runs on a dedicated server. The memory available for the JMeterEngine is limited to 1.5GB, triple the amount provided by the default configuration. Because this configuration uses one server per one JMeterEngine it requires additional Computing Hours compared to the default configuration.

There are a few pre-requisites in order to run tests. This page provides compatibility tests for making sure all is in place and you are able to run tests.

JavaScript is Enabled

Please press the button below to test whether JavaScript is enabled. If the text JavaScript is Disabled changes to JavaScript is Enabled the JavaScript on your local machine is enabled and you can proceed to the next compatibility test.

Java is Installed

Test and update your Java installation by visiting this link and follow the instructions.

Outbound port 22 is open

As we are using SSH (port 22) to communicate with the remote servers, follow the instructions below to verify that outbound port 22 is open. If you can successfully open the Open-Office application accessible by the link below, the outbound port 22 is open on your local machine and you can proceed to the next compatibility test. If for some reason you are unable to access this application, please check that outbound port 22 is open.

Open-Office Application

Explanation: Once you press on the link above, you are directed to NOMACHINE test-drive page. Click the “Open-Office” icon.

 You are now directed to a page with an applet. Click the “Continue” button.

This should open the Open-Office application to indicate that the remote connection is enabled. If for some reason, the applet does not appear or the connection is cut off and the application does not open, please contact our support team for further instructions.

You can follow the very nicely written instructions as they appear on IBM web site: http://www.ibm.com/developerworks/opensource/library/os-jmeter/

You can follow the very nicely written instructions as they appear on IBM web site: http://www.ibm.com/developerworks/data/library/techarticle/0303bhogal/03...

You can follow the very nicely written instructions as they appear on Oracle web site: http://www.oracle.com/technology/pub/articles/dev2arch/2006/08/jmeter-pe...

If you are using Cognos software, IBM recommends to use JMeter for stability tests. You can use JMeter in the Cloud service by CloudIntelligence to upload the JMeter test-scripts and immediately start executing stability and/or load tests over your Cognos installation.

Additional Resources:
JMeter Cognos How-To
Cognos 8 JMeter Stability Test
JMeter in the Cloud

You can view all public DNS names involved in a test session in the Network Information tab at the bottom of the page:

This can be used to configure your firewall to allow access from the specified domain names. Please note that each test session uses different domain names. Once all servers are launched all public DNS names are displayed at the Network Information tab.

LoadIntelligence enables you to add additional user properties file to the launched servers. Add your JMeter properties to the default “user.properties” file which is defined by the JMeter property and use the regular file upload mechanism to upload the file.

LoadIntelligence will identify the user.properties file and replace the JMeterEngines and console user.properties file with the newly uploaded file. Similarly, system.properties can be uploaded to update system properties.

Users that develop and/or enhance their JMeter components can now upload their own JMeter components and add-ons.

To do this, jar your JMeter components and use the regular file upload mechanism to upload the jar file. LoadIntelligence will identify the uploaded jar files and will automatically copy them to JMETER_HOME/lib/ext.

In order to save computing hours, you may want to terminate all of the load servers, but keep the GUI console open. In this case the load servers are terminated except for the server running the JMeter GUI console, therefore the load results are still available for viewing and uploading to the website. To terminate only the load servers, click on Terminate All But Console button on the right-hand menu:

When you no longer need to view the JMeter GUI console, click on Terminate All button in order to terminate the remaining server.

LoadIntelligence enables you to control the hosts file of the launched servers. As a result, you are able to add additional local DNS entries as if you were editing the hosts file locally.

One important use for this feature is to have a permanent DNS resolution for certain DNS names. This can reduce the number of times LoadIntelligence calls on your DNS for name resolution.

To do this just use the regular file upload mechanism to upload a hosts file. The new hosts file should include at least one line of the form: 127.0.0.1 localhost. LoadIntelligence will identify the hosts file and replace the servers’ hosts file with the newly uploaded file.

LoadIntelligence complements the Load Runner application by enabling Load Runner's users to increase the number of concurrent virtual users while leveraging pre-written test scripts. These new virtual users can be simulated from various locations in the world.

For example, assuming an organization bought Load Runner licenses for 1,000 virtual users. This organization can use LoadIntelligence to simulate additional 2,000 virtual users in multi geographic regions.

Please contact us for further information.

A reservation is an EC2 term used to describe a group of servers launched at the same time under the same web service request. EC2 assigns a unique reservation name for all of these servers. All operations under LoadIntelligence are performed on reservations. Reservations are launched, rebooted and terminated.

You can run the same test several times, each time choose a different location in the world. In Add/Edit Test page, “Test Properties” section, selected the desired location in “Load origin location” field:

In the JMeter application, in the results table view, click on the “Save Table Data” button located under the table. In the open dialog, name your results csv file (or png file for saving graph’s results), do NOT use spaces in the file name and save it in the default location:

Your saved files can be downloaded using the Files tab.

You can monitor each server’s CPU and bandwidth data during the load. Go to the test page, at the bottom there is a table with tabs, each tab is named after a server, choose the server’s name you wish to track and you will be displayed with the server’s vital signs:

Note that the display is automatically updated every 30 seconds. You can also go over this information after the reservation had terminated, by downloading the zip file of all the reservation logs:

You are not limited by time, once you have launched a reservation, you can use the servers as long as you need. When you are done with your tests you should terminate the reservation.

Use the parameter ${__P(InstanceID)}. Each JMeterEngine uses a distinct parameter ${__P(InstanceID)}. Please note that the "InstanceID" values start from 1 for the JMeterEngines. The value of "InstanceID" for the JMeter console is 0.

See this FAQ for more information.

Yes. Once all servers are ready and the remote JMeter application opened, go to the Top Bar, under "Run" menu select "Remote Start". All available JMeterEngines are displayed and you can click on the one you want to start:

The same can be done for manually stopping a JMeterEngine (by choosing a JMeterEngine from "Remote Stop" list).

When running the JMeter remote application, you need to click on “Remote Start All” in order to activate the remote JMeterEngines:

Note that you will need to click on “Remote Stop All" for stopping all remote JMeterEngines.

The remote application requires your approval for connecting to the load servers, due to Amazon’s security standards. There is probably an open dialog box waiting for your approval:

Minimize all open windows , or use Ctrl+Alt to look for that open pop-up. Once approved the JMeter application will open and you can continue with your test.