Performance has greatly degraded on a pair of 64GB M4 drives

SOLVED
Binary Boss

Re: Performance has greatly degraded on a pair of 64GB M4 drives

Are both m4s using the same firmware version?

 

The before and after pictures show different alignment - so what's changed? It wouldn't change itself... If you imaged/secure erased/re-imaged, the alignment shouldn't change, assuming you're using a proper program. Do you have an "after" picture on your original alignment as shown in the first picture...?

 

Something's not adding up for me here! Smiley Very Happy

Silicon Sultan

Re: Performance has greatly degraded on a pair of 64GB M4 drives

i guess i don't see an advantage to the partitioning. wouldn't single partition/multiple folders be just as effiecent?

as it is, your partitioning may be the issue as if your c is benching normal...and yes, there is no "before" of the e partition... is the e partition using a different cluster size?

Highlighted
Kilobyte Kid

Re: Performance has greatly degraded on a pair of 64GB M4 drives

I have done a bunch more testing to try to isolate the problem, and am now out to sea.

 

First, I broke the raid, and tested one of the drives as a normal drive.  I tested wiht AS-SDD on partition sizes of 8GB and 59GB (the maximum integral GB partition for the drive), and for each of these two partition sizes I tested with cluster sizes of 4K, 8K, and 16K.  All of these choices ran fine, producing 4K-64Thrd numbers of  roughly 150 MB/s read and 90 MB/s write.  So partition size and cluster size did not seem to have any material effect on the results.  I then ran one of these cases on the other M4 drive, and got the same good result (it did not seem necessary to run all of the tests on the second drive).

 

Next, I rebuilt the raid, and tried 4K, 16K, and 64K stripes on a partition of 119 GB (the maximum integral GB partition for the raid array).  Here, too, I got good numbers, roughly 270 MB/s read and 180MB/s write on the 4K-64Thrd tests.  On the 4K stripe raid I also tried the 8GB partition and 59 GB partition to mirror the single-drive tests above, and got the same numbers as for the rest of the raid tests (270 MB/s and 180 MB/s read and write).  So it looks as if the stripe size is not causing the performace to degrade, either.

 

Note that all of the above tests were performed on freshly allocated and formatted partitions.

 

Then, on the 64K stripe raid, I restored my imaged data, and tested that.  Immediately the results were back to less than 20 MB/s write on the 4K-64Thrd test.  This is on the same physical disk setup which a moment before had yielded 180 MB/s.  It made no difference which of my restored partitions I tested; the numbers were in this less-than-20 area.

 

I tried deleting all the data (including the recycle bin) on a couple of the partitions so that the partitions were completely empty.  AS-SDD still gave me 20 MB/s on the writes.  I then tried formatting the empty partitions -- there was nothing on them but I ran format anyway.,  And -- who knows why -- the data went back to the 180 MB/s write ballpark.  I tried copying all of the data that had been on the partition back onto the partition, and the rate still stayed in the fast lane.

 

I ran this specific case again:  one 25GB partition freshly restored with my data and AS-SDD shows the 20 MB/s number, all data deleted and AS-SDD shows the 20 MB/s number, simple format of that partition and AS-SDD shows the 180 MB/s number, copy all data back and AS-SDD shows the 180 MB/s number.

 

(These last paragraphs make me wonder about a case I did not run, and am too weary to run now.  Break the raid again, restore a couple of partitions onto the single drives, and test them there.  Will I see the 20 MB/s number or the 180 MB/s number?  All of the single-drive testing was done, as mentioned, with newly allocated and formatted partitions, and we see these newly allocated and formatted partitions running nicely in raid mode as well.)

 

So, as I said at the top, I am perplexed. 

 

 

Kilobyte Kid

Re: Performance has greatly degraded on a pair of 64GB M4 drives

Further investigation has revealed the reason why the AS-SDD 4K and 4K-64Thread write benchmarks are slow on some partitions but not on others. I have been testing with partitions that are aligned on 1-megabyte binary boundaries and that have cluster sizes of 4K, 8K, 16K, or 32K.  The reason is this:

 

  • even when a partition is aligned on 1-megabyte binary boundary, the clusters in the partition are not necessarily aligned on any boundary at all -- they may start on a sector with any alignment at all with respect to the partition start sector and therefore they may start on a sector with any alignment at all with respect to the physical drive

Here are the details.  I used the excellent Process Monitor tool from Sysinternals http://technet.microsoft.com/en-us/sysinternals to see just what was happening when AS-SDD executes the 4K-64Thread write benchmark.  After initializing, the benchmark simply writes 4K blocks to random points of 64 different files, as you would expect.  Further, each of these 4K blocks is aligned at a 4K boundary within its file:  bytes 0-4091, for example, or 4092-8191; always 4K, always 4K-aligned in the file.  I verified that when I was seeing slow performance on this benchmark there was no other I/O activity going on (and no other acitivity at all, really, except for minor noise).

 

Then it occured to me to ask whether the clusters comprising the file were themselves actually aligned to a cluster boundary?  Whether the 8 sectors in a 4K cluster for example were aligned so that its first sector was also on a 4K byte boundary in the partition and on the dirve? A quick test using another Sysinternals tool, DiskMon, revealed that this need not be true:  on partitions that showed slow performance, the 8 sectors started on a sectors aligned not on 4K boundaries but on 4K+512, or 4K+1024, or other non-4K-aligned boundaries.  So,.although the partitions were aligned, a 4K write to a single 8-sector cluster might actually require writing to two different 1-megabyte blocks (or 256K blocks or whatever power of two is relevant to the internal pages of the M4 SSD or to the internal stirpe size used for buffering by the RAID).  The I/O activity to the partition is for all practical purposes occurring just as if the partiiton were not aligned, because the internals of the partition are not aligned.

 

I then used the excellent HxD hex editor //http://mh-nexus.de/en/ (which is able to open partitions and physical drives for easy inspection) to see what was going on in the partition structure.  It turns out that FAT32 disks do not require clusters to be aligned at all.  They are all the same size, and follow one another contiguously, but the first cluster may start at an arbitrary offset within the partition.  And that is what is happening to my partitions.  Using info on the FAT32 structure from Microsoft http://technet.microsoft.com/en-us/library/bb457122.aspx the first cluster is located in the partiton at a sector computed as the sum of (a) the first sector in the partition, (b) the offset from the first sector in the partition to the FAT tables, and (c) the size of one FAT table times the number of copies of the FAT tables (two in my cases).  On my partitions (a) is always a nice megabyte-binary number, but (b) can be a pure odd number with no relation to cluster size at all, and (c) is likewise an arbitrary sectory count (but always an even number).

 

So, for the purposes of my original query -- why is performance so degraded -- the answer is that even though my partitions are aligned externally they are in real effect actually unaligned because of their internal cluster (mis)alignment.  Who ever would have thought that a partition creation program would create misaligned internal structures?  But here is how this likely came about.

 

I began with partiitons that were created on spinning drives, and therefore they were allocated with cylinders and tracks defining their borders and extents.  This invloves strange numbers like 63 and 255, and nothing is aligned on 4K boundaries.  I imaged these partitions using Paragon's free backup and restore program http://http://www.paragon-software.com/home/br-free/index.html and then, after creating megabyte-aligned partions using this Paragon program, I restored the partitions from the image into the aligned partitions.  But Paragon apparently does not align the clusters as part of this process.  Freshly created partitions seem to have 4K-(at least)-aligned clusters, but I see no guarantee or specification for the alignment.

 

So I will reallocate and copy all of the partitions, creating them freshly rather than by restoring a cylinder-track-oriented image.  But for those of you who answer queries from users, this is just another thing to check: yes, the partition is aligned, but are the clusters?  All the discussion on all the boards I read was about partiiton alignment -- how to get it and how to preserve it and why this software or that software doesn't preserve alignment -- and not one mention that I saw of clusters.  Who woulda thunk.