On Linux you can look into the /proc/diskstats file, which shows something like:

 251       0 vgca0 30273954 0 968968610 416767 122670649 0 8492649856 19260417 0 19677184 220200747

where 8492649856 is the number of sectors written since the reboot (sector is 512 bytes).

Now you can say that we may take /proc/diskstats stats with the 1h interval, and it will show write how many bytes per hour we write, and in such way to calculate the potential lifetime.
This will be only partially correct. There is such factor as Write Amplification, which is very well described on WikiPedia, but basically SSD cards, due an internal organization, write more data than it comes from an application.
Usually the write amplification is equal or very close to 1 (meaning there is no overhead) for sequential writes and it gets a maximum value for fully random writes. This value can be 2 – 5 or more and depends on many factors like the used capacity and the space used for an over-provisioning.

Basically it means you should look into the card statistic to get an exact written bytes.
For Virident FlashMAX it is

vgc-monitor -d /dev/vgca  | grep writes
                                 379835046150144 (379.84TB) (writes)

Having this info let’s take look what a lifetime we can expect under a tpcc-mysql workload.
I put 32 users threads against 5000W dataset (about 500GB of data on the disk) during 1 hour.

After 1 hour, /proc/diskstat shows 984,442,441,728 bytes written, which is 984.44GB and the Virident stat shows 1,125,653,692,416 bytes written, which is 1,125.65GB
It allows us to calculate the write amplification factor, which in our case is
1,125,653,692,416 / 984,442,441,728 = 1.143. This looks very decent, but remember we use only 500GB out of 1400GB, and the factor will grow as we fill out more space.

Please note we put a quite intensive write load during this hour.
MySQL handled 25,000 updates/sec, 20,000 inserts/sec and 1,500 deletes/sec, which corresponds to
write throughput 273.45MB/sec from MySQL to disk.

And it helps to calculate the lifetime of the card if we put such workload 24/7 non-stop.
15PB (of total writes) / 1125.65GB (per hour) = 13,325.634 hours = 555.23 days = 1.52 years

That is under non-stop tpcc-mysql workload we may expect the card will last 1.52 years. However, in real production you do not have an uniform load every hour, so you may base your estimation on daily or weekly stats.

Unfortunately there is no easy way to predict this number until you start workload on the SSD.
You can take look into /proc/diskstat, but
1. There is write amplification factor which you do not know
and 2. A throughput on regular RAID is much less than on SSD and you do not know what your throughput will be when you put workload on SSD.

At Virident’s request, I performed an evaluation of their MLC card to assess how it handles MySQL workload. As I am very satisfied with the results, I wish to share my findings in this post.

But first, I wish to offer an overview of the card.

Virident FlashMax Cards are available in 1TB and 1.4TB usable capacities (the models names are M1000 and M1400)
These specified sizes are already available for end users.
I evaluated M1400 (1.4TB size) model, which I will discuss:

Because Virident has competition in the SSD market, they have stated their goals to distinguish themselves from their competitors:

• Stability of performance: That is to minimize variations in throughput
• Better response times: This is very important for database performance and I appreciate that Virident has made this a priority.
• Performance at full capacity: As we know, SSD-based cards have special characteristics; the throughput declines when space utilization increases. Virident’s design/programming minimizes this decline.
• RAID5 on the card: The card comes with RAID5 support on the card to give better protection.

To deal with a throughput decline, all Flash cards have reserved space. The 1.4TB card, that I have, internally holds 2TB worth of space.

This additional space is used for two purposes:

1. To amortize write-intensive workloads, by using additional space.
2. To have replacements for failed MLC modules. When one MLC module fails, it is marked as unused, and gets replaced by one from the pool of reserved modules.

Internally, Virident uses 25nm Intel NAND Flash MLC modules, this is the same technology that Intel uses for the Intel SSD 320 cards. 25nm modules allow the user a greater capacity, Physically you can place
more GBs into a given area. However, the drawback is that 25nm has worse reading and writing latencies, compared to previous generations. However, I have yet to determine how this affects MySQL workloads.

Virident has provided the following price list:

• M1000 (1000GB Usable) – $13,000
• M1400 (1400GB Usable) – $18,200
• This amounts to $13/GB

Second, it is important to compare the performance of Virident FlashMAX MLC with available competing solutions.
It is fair to say Fusion-io ioDrive Duo 1.28TB MLC is the most well-known and most advanced competitor in the market.
I had a chance to administer a head-to-head comparison of sysbench and tpcc-mysql workloads between FlashMAX 1.4TB and ioDrive Duo 1.28TB.

It is important to highlight that Fusion-io ioDrive Duo is based on 34nm NAND technology, which is a full generation behind the 25nm NAND. However at this point, I have no access to Fusion-io ioDrive2, which is based on 25nm NAND.
Another important factor is that ioDrive Duo is actually two cards visible in the OS, and the user needs to use a software RAID. For Virident all 1400GB shows up as one single drive so no software RAID is necessary.

To compare performances I ran sysbench oltp and tpcc-mysql benchmarks. I will present the results
for sysbench oltp (with full report available later) below, and the results for tpcc-mysql in a followup post.

For sysbench, I used our multi-tables sysbench implementation with 256 tables and 10,000,000 rows each. This is a total of around 630GB of data, which allows one to adequately fill both cards in comparison.

Some hardware used in benchmarks include:

• Server: Cisco UCS C250, running Oracle Linux 6.1 and Percona Server 5.5.15
• Client: HP ProLiant DL380 G6, sysbench v5

Of course, our Percona Server was optimized for Flash cards, with variations for two settings.
I tested combinations of innodb_buffer_pool_size=120GB, 174GB and innodb_flush_log_at_trx_commit=1, 2.

The results in this post are for case innodb_buffer_pool_size=174GB and innodb_flush_log_at_trx_commit=1

As in all my recent benchmarks, I use long runs of 1 hour each with measurements every 10 seconds. This methodology allows me to observe trends and the stability of the performance on graphs.

The first graph represents throughput in transactions per second for different amounts of user threads (more is better). More concentrated dots represent less variance and better stability of throughput.

A tabular format, for throughput I use a median of measurements for last 1800 seconds in each run:

In order to examine the details of how throughput varies we have taken 32 threads and examined the timeline graph for each one:

While you can see that with Virident FlashMAX we have a pretty stable line of around 975 tps, the Fusion-io ioDrive Duo has a variance of 700-800 tps.

My conclusions are as follows:

• It is great to see another player on MLC Flash cards market.
• It is also great that Virident focuses on stability of performance for competitive advantage.
• Beside stability, we also see better throughput in MySQL using the Virident FlashMAX card for every thread count. On 32-64 threads we have about a 35-40% advantage of using Virident FlashMAX.

DISCLOSURE: This review was done as part of our consulting practice for which we compensated by Virident. However, this review was written independently of Virident, and reflects our opinion of this product.

Full report is available in PDF

For tests we used tpcc-mysql package, which generates TPCC-like workload on MySQL systems.

• Server hardware: Dell PowerEdge R815
• Storage: Fusion-io ioDrive Duo 640GB MLC. Fusion-io driver version: 2.3.1 build 123; Firmware v5.0.7, rev 101971
• Software: Percona Server 5.5.15
• Client hardware: IBM x3650

Fusion-io ioDrive Duo 640GB MLC card was provided by Fusion-io.

More details on hardware and software configuration are in full report

tpcc-mysql tests were run for following combinations:

2400W, big buffer pool

• 1 MySQL instance, 2400 warehouses (220GB of data), 120GB buffer pool
• 2 MySQL instances, 1200 warehouses (110GB of data) each, 60GB buffer pool each
• 4 MySQL instances, 600 warehouses (55GB of data) each, 30GB buffer pool each

2400W, small buffer pool

• 1 MySQL instance, 2400 warehouses (220GB of data), 64GB buffer pool
• 2 MySQL instances, 1200 warehouses (110GB of data) each, 32GB buffer pool each
• 4 MySQL instances, 600 warehouses (55GB of data) each, 16GB buffer pool each

1200W, big buffer pool

• 1 MySQL instance, 1200 warehouses (110GB of data), 120GB buffer pool
• 2 MySQL instances, 600 warehouses (55GB of data) each, 60GB buffer pool each

1200W, small buffer pool

• 1 MySQL instance, 1200 warehouses (110GB of data), 64GB buffer pool
• 2 MySQL instances, 600 warehouses (55GB of data) each, 32GB buffer pool each

The purpose of different combination of data and memory sizes, was to check
how data/memory ratio affects results.

We used 48 user sessions and we performed 2700 sec long run, gathering data for New Order Transaction each 10 seconds.

That is, for each set of user sessions, we take 270 throughput measurements.

Based on this, we constructed Median Throughput for last 900 sec, to avoid warm-up influence on results.

In conclusion I can highlight:

• Running multiple instances shows good improvement in throughput. 1.4x-1.8x for 2 instances and 1.6-2.4x for 4 instances.
• If you have sharding environment which allows you separate database into multiple instances you may try 2-4 instances setup to get better overall throughput from your MySQL setup

Recently Fusion-io announced new ioDrive2 card with new driver, which promises even better performance for MySQL. I am looking forward to test it.

Disclaimer: This benchmark is sponsored by Fusion-io, but this post is totally independent and fully reflects our opinion.

Somewhat each time on Intel 320 SSD I was getting different write performance, so it made me looking into this with details.

So let’s run experiment as in previous post, this is sysbench fileio random write on different file size, from 10GiB to 140GiB with 10GiB step. I use ext4 filesystem, and I perform filesystem format before increasing filesize.

The results are pretty much as in previous post, the throughput drops as we increase filesize:

However, there is when interesting stuff begin. Now when we run the same iterations again, the result will look like:

As you see, second time the throughput is much worse, even on medium size files. Just after 50GiB size, throughput gets below 40MiB/sec And this is with the fact, that I perform filesystem format before each run.

This leads me to conclusion that write performance on Intel 320 SSD is decreasing in time, and actually it is quite unpredictable in each given point of time. Filesystem format does not help, and only secure erase procedure allows to return to initial state. There are commands for this procedure for reference.

hdparm --user-master u --security-set-pass Eins /dev/sd$i
hdparm --user-master u --security-erase Eins /dev/sd$i


Discussing this problem with engineers working with Intel 320 SSD drives I was advised to use artificial space provisioning, about 20%. Basically we create partition which takes only 80% of space.

So let’s try this. The experiment the same as previously, with difference that I use 120G partition, and max filesize is 110GiB.

You can see that throughput in first iteration is basically the same as with full drive, but second iteration performs much better. Throughput never drops below 40MiB/sec, and stays on about 50MiB/sec level.

So, I think, this advise to use space provisioning is worth to consider if you want to have some kind of protection and maintain throughput on some level.

Raw results and used scripts as always you can find on our Benchmarks Launchpad

Intel 320 SSD appears to be a popular drive with a quite acceptable price.
I wrote about write performance of these cards, and now let’s take look on a random read workload.

I used a Cisco UCS C250 as base hardware, comparing in it:

• regular RAID10 over 8 SAS 2.5 disks
• single Intel 320 SSD directly attached to a HighPoint RocketRAID 2300
• two Intel 320 SSD in hardware RAID0 mode, attached to a LSI SAS9211-4i controller

For simulating the workload I used sysbench’s fileio random reads. Scripts and raw results available on Launchpad.

Let’s see throughput results:

And response times:

As conclusion, this card provides great read performance. A single card provides 5-8x better throughput and 10-14x better response time. Striping helps to increase throughput in 8-10x and response time in 10-22x.

While there are questions about write performance (see my previous post), I think this card is very suitable for read-intensive tasks, where you can expect significant improvements.

Follow @VadimTk

I was advised that new drivers and new firmware for FusionIO cards improve performance and stability and it is recommended to review results I’ve got about year ago.

Using the same methodology and the same box as for Intel 320 SSD, I run random writes benchmarks for FusionIO 320GB MLC card (I do not have the card I had year ago on hands).

Information about system, FusionIO drives are raw results are on Benchmarks Launchad

First graph is to show timeline for different filesizes. Benchmark starts just after formatting card and filesystem and runs around 1 hour with measuring throughput each 10 sec.

Interesting to see the same pattern as for Intel 320 SSD: the throughput starts at max, then drops down and after some peak stabilizes.

1500 sec seems enough to get stable line for all filesizes. If we take slice of data after 1500 sec and build summary space->throughput graph, it looks like:

So we still have decent declining line. The throughput drops from 500MiB/sec at peak to 110-120MiB/sec at full capacity.

And to have some fun with R/ggplot2 graphs, let’s build graph to compare FusionIO and Intel 320 SSD (with results from previous post)

Follow @VadimTk

While I like performance provided by PCI-E cards like FusionIO or Virident tachIOn, I am often asked about SATA drives alternatives, as price of PCI-E cards often is barrier, especially for startups. There is wide range of SATA drives on market, and it is hard to pick one, but Intel SSD are probably one of most popular, and I’ve got pair of Intel 320 SSD 160GB to play with it.

Probably most interesting characteristic for SSD for me is Random Write throughput in correlation with file size , as it is known that the write throughput declines when you use more space.

In this post I will test (using sysbench fileio) single Intel 320 SSD card with different filesize ( from 10 to 140 GiB, with step 10GiB). Filesystem is XFS and IO blocksize is 16KiB.

I posted all scripts and results on our Launchpad project where you can find

• Script to run benchmark
• System summary
• Information from Intel 320 SSD card
• Raw results
• Helper R script used to build graphs

I used next methodology for testing: format xfs, run 1 hour random write test with measuring throughput each 10 sec.

The results are bit tricky to analyze, as throughput performs in this way (for 100GiB filesize)

You can see that just after format throughput starts with 80MiB/sec, then drops to 10MiB/sec and after about half of hour stabilizes on 30MiB/sec level.

We can build the same graph (time -> throughput) for all filesizes we have:

where you can see that throughput drops from 100 MiB/sec for 10GiB file to 15MiB/sec for 140GiB file.

For reference I added result from similar benchmark for RAID10 over 8 regular spinning SAS 15K disks, which is around 23MiB/sec.

From graph we see that all results are stabilized after 2500 sec, and if we get slice of data after 2500 sec, the summary graph ( size -> throughput) looks like:

This graph allows to get idea what is throughput for given filesize much easier.

E.g. for 70GiB files, we have 40MiB/sec and for 120GiB file, it is 20MiB/sec.

Some conclusions from these results:

• Intel 320 SSD performance is affected by amount of used space. The more space used – the worse performance
• Throughput may drop very intensively, e.g. from 10GiB to 20GiB it drops by 20%
• When you run benchmark on your own, take into account time needed to get stabilized result. It may take over half of hour for some cases

In final I want to give credit to R projects and ggplot2 which are very helpful for graphical analyzing of data.

Follow @VadimTk

This is to follow up on my previous post and show the results for MySQL 5.5.8 and Percona Server on the fastest hardware I have in our lab: a Cisco UCS C250 server with 384GB of RAM, powered by a Virident tachIOn 400GB SLC card.

To see different I/O patterns, I used different innodb_buffer_pool_size settings: 13G, 52G, an 144G on a tpcc-mysql workload with 1000W (around 100GB of data). This combination of buffer pool sizes gives us different data/memory ratios (for 13G – an I/O intensive workload, for 52G – half of the data fits into the buffer pool, for 144G – the data all fits into memory). For the cases when the data fits into memory, it is especially important to have big transactional log files, as in these cases the main I/O pressure comes from checkpoint activity, and the smaller the log size, the more I/O per second InnoDB needs to perform.

So let me point out the optimizations I used for Percona Server:

• innodb_log_file_size=4G (innodb_log_files_in_group=2)
• innodb_flush_neighbor_pages=0
• innodb_adaptive_checkpoint=keep_average
• innodb_read_ahead=none

For MySQL 5.5.8, I used:

• innodb_log_file_size=2000M (innodb_log_files_in_group=2), as the maximal available setting
• innodb_buffer_pool_instances=8 (for a 13GB buffer pool); 16 (for 52 and 144GB buffer pools), as it is seems in this configuration this setting provides the best throughput
• innodb_io_capacity=20000; a difference from the FusionIO case, it gives better results for MySQL 5.5.8.

For both servers I used:

• innodb_flush_log_at_trx_commit=2
• ibdata1 and innodb_log_files located on separate RAID10 partitions, InnoDB datafiles on the Virident tachIOn 400G card

The raw results, config, and script are in our Benchmarks Wiki.
Here are the graphs:

13G innodb_buffer_pool_size:

In this case, both servers show a straight line, and it seems having 8 innodb_buffer_pool_instances was helpful.

52G innodb_buffer_pool_size:

144G innodb_buffer_pool_size:

The final graph shows the difference between different settings of innodb_io_capacity for MySQL 5.5.8.

Small innodb_io_capacity values are really bad, while 20000 allows us to get a more stable line.

In summary, if we take the average NOTPM for the final 30 minutes of the runs (to avoid the warmup stage), we get the following results:

• 13GB: MySQL 5.5.8 – 23,513 NOTPM, Percona Server – 30,436 NOTPM, advantage: 1.29x
• 52GB: MySQL 5.5.8 – 71,774 NOTPM, Percona Server – 88,792 NOTPM, advantage: 1.23x
• 144GB: MySQL 5.5.8 – 78,091 NOTPM, Percona Server – 109,631 NOTPM, advantage: 1.4x

This is actually the first case where I’ve seen NOTPM greater than 100,000 for a tpcc-mysql workload with 1000W.

The main factors that allow us to get a 1.4x improvement in Percona Server are:

• Big log files. Total size of logs are: innodb_log_file_size=8G
• Disabling flushing of neighborhood pages: innodb_flush_neighbor_pages=0
• New adaptive checkpointing algorithm innodb_adaptive_checkpoint=keep_average
• Disabled read-ahead logic: innodb_read_ahead=none
• Buffer pool scalability fixes (different from innodb_buffer_pool_instances)

We recognize that hardware like the Cisco UCS C250 and the Virident tachIOn card may not be for the mass market yet, but
it is a good choice for if you are looking for high MySQL performance, and we tune Percona Server to get the most from such hardware. Actually, from my benchmarks, I see that the Virident card is not fully loaded, and we may benefit from running two separate instances of MySQL on a single card. This is a topic for another round.

(Edited by: Fred Linhoss)

In this regard, Virident claims that tachIOn cards provide “Sustained, predictable random IOPS – Best in the Industry”. Virident generously provided me model 400GB, and I ran the benchmark using the
same methodology as in my experiment with FusionIO, which was stripped for performance. Also using my script, Virident made runs on tachIOn 200GB and 800GB model cards and shared numbers with me ( to be clear I can certify only numbers for 400GB card, but I do not have reasons to question the numbers for 200GB and 800GB, as they corresponding to my results).

The benchmarks was done on Cisco UCS C250 box and raw results are on Benchmarks Wiki

Visually, the drop is not as drastic as it was in the case using FusionIO, but let’s get some numbers.
I am going to take the performance numbers at the points where the available space of the card is 1/3, 1/2, and 2/3 filled, as well as at the point where the card is full. Then I will compute the ratio of each of those IOS numbers to the IOS at the 1/3 point.

**For the 400GB tachIOn card:**

That is, at the 2/3 point, the 400GB card is slower by 29% than at the 1/3 point, and at full capacity it is slower by 57%.

Observations from looking at the graph:

* You can also see the card never goes below 400MB/sec, even when working at full capacity. This characteristic (i.e., high throughput at full capacity) is very important to know if you are looking to use an SSD card as a cache layer (say, with FlashCache), as, usually for caching, you will try to fill all available space.
* The ratio between the 1/3 capacity point and full capacity point is much smaller compared with FusionIO Duo (without additional spare capacity reservation).
* Also, looking at the graphs for Virident and comparing with the graphs for FusionIO, one might be tempted to say that Virident just has a lot of space reserved internally which is not exposed to the end user, and this is what they use to guarantee a high level of performance. I checked with Virident and they tell me that this is not the case. Actually from diagnostic info on Wiki page you can see: tachIOn0: 0×8000000000 bytes (512 GB), which I assume total installed memory. Regardless, it is not a point to worry about. For pricing, Virident defines GB as the capacity available for end users. So, a competitive $/GB level is maintained, and it does not matter how much space is reserved internally.

Now it would be interesting to compare results with results for FusionIO. As cards have different capacity I made graph which shows
throughput vs percentage of used capacity for both cards, FusionIO 320GB Duo SLC and Virident 400GB SLC

In conclusion:
* On single Virident card I see random write throughput close or over 1GB/sec in with low space usage and it is comparable with throughput I’ve got on stripped FusionIO card. I assume Virident maintain good level of parallelism internally.
* Virident card maintains very good throughput level in close to full capacity mode, and that means you do not need to worry ( or worry less) about space reservation or formatting card with less space.

To see how performance drops with decreasing free space, let’s run sysbench fileio random write benchmark with different file sizes.

For test I took FusionIO 320 GB SLC PCIe DUO™ ioDrive card, with software stripping between two cards, and there if graph how throughput depends on available free space ( the bigger file – the less free space)

The system specification and used scripts you can see on Benchmark Wiki

On graph you can see two line ( yes, there are two lines, even they are almost identical).
First line is result when FusionIO is formatted to use full capacity, and second line is for case when I use additional space reservation ( 25% in this case, that is 240GB available). There is no difference in this case, however additional over-provisioning protects you from overusing space, and keeps performance on corresponding level.

It is clear the maximal throughput strongly depends on available free space.
With 100GiB utilization we have 933.60 MiB/sec,
with 150GiB (half of capacity) 613.48 MiB/sec and
with 200GiB it drops to 354.37 MiB/sec, which is 2.6x times less comparing with 100GiB.

So returning to question how to run proper benchmark, the result significantly depends what percentage of space on card is used, the results for 100GiB file on 160 GB card, will be different from the results for 100GiB file on 320 GB card.

Beside free space, the performance also depends on garbage collector algorithm by itself, and the card from different manufactures will show different results. Some new coming cards make high performance in case with high space utilization as competitive advantage, and I am going to run the same analysis on different cards.