Improper rebuild is a mistake customers often make when trying to solve array problems on their own. This issue is complex enough to confuse a data recovery engineer. It is important to understand exactly what is happening to the data and how it affects the chances of a successful data recovery.
We will look at the most popular level — RAID-5. We will consider how the processes of initialization, correct and incorrect rebuild are performed for it. Furthermore, we will discuss whether it is possible to recover data after an incorrect rebuild.
There can be many variants of incorrect rebuild for RAID-5, in this article we will limit ourselves to just one of them — when the new and old configurations are the same. This is the common case in practice and the easiest to explain and recover data from. Once we have studied it, we can move on to more complex cases in the following articles.
We assume that you have studied the previous article on RAID-1 and understand what we mean by initialization, rebuild, and the nature of the “improper rebuild” problem. So let’s go straight to the specifics of these processes for RAID-5.
Difference between proper and improper rebuild
Redundancy in RAID-5
Let’s consider a three-drive RAID-5 LS (Left Synchronous) scheme:
A template for a RAID-5 LS of three drives.
Each row has a block that stores the XOR of the other blocks:
C0 = A0 xor B0,
B1 = A1 xor C1,
A2 = B2 xor C2
The data is divided into blocks. Each row contains data blocks. However, there is a special block that stores the result of the XOR operation calculated for all data blocks in that row. We will call this block the “XOR block”. There are several RAID-5 schemes, but for the purposes of this article, the differences between them are not important.
With XOR blocks, we can recover the data of any one failed member. For example, if block C0 = A0 xor B0, then:
- A0 = B0 xor C0
- B0 = A0 xor C0
- C0 = A0 xor B0 (by definition).
Similarly for other rows — any member can be recovered using the XOR operation and the remaining members. And the same rule applies to RAID-5 with more members.
This brings us to a property of any properly functioning RAID-5: the XOR blocks must be valid. The result of XORing data blocks must equal the value stored in the XOR block.
RAID-5 Initialization
If we take arbitrary drives and try to create RAID-5, we cannot be sure that the XOR blocks are valid. Therefore, RAID-5 arrays need to be initialized.
How do we make the XOR blocks valid? The obvious solution is to calculate the XOR of the data blocks and write the result into the appropriate XOR block. This is what most RAID controllers do.
RAID 5 initialization
XOR block calculation and update is performed:
A0 XOR B0 -> C0
A1 XOR C1 -> B1
B2 XOR C2 -> A2
That is, during initialization, the data is partially updated on all members.
Rebuild After Degradation
If one of the members fails, RAID-5 degrades — data is still available (the data of the missing drive is computed on the fly), but reliability is no longer guaranteed.
To restore the properties of the array, the failed drive must be replaced with a new one. Will it be enough to simply add a new drive to the array? No. The new drive does not contain correct data or valid XOR blocks. It is necessary to perform a rebuild, a procedure that writes the correct data to the new drive. How to get the correct data? It’s simple — calculate the XOR of the old members.
RAID-5 rebuild (restore member C):
XOR the data of members A and B and write the result to NewC
A0 xor B0 -> NewC
A1 xor B1 -> NewC
A2 xor B2 -> NewC
This means: During a proper rebuild, only the new drive will be updated.
Improper rebuild
Let’s take a look at how the data changes when an initialization is done instead of a (proper) rebuild. RAID-5 has many parameters, for simplicity we will assume that none of them have changed.
Initialization with new member in RAID-5:
XOR block recalculation is performed:
A0 xor B0 -> NewC0 (correct)
A1 xor NewC1 -> B1 (wrong)
B2 xor NewC2 -> A2 (wrong)
The initialization will recalculate and update all XOR blocks. In this case:
- The data blocks on the new drive will remain unchanged, i.e., they will continue to store unneeded data.
- The XOR blocks on the old drives will be recalculated based on the wrong data, so they will be incorrect.
Damage Analysis
Let’s understand how these damages affect the logical RAID image — what the data on the assembled RAID will look like.
Data of the RAID assembled from drives A, B, and NewC.
Every 6 blocks contains 2 bad blocks.
For a three-disk RAID-5, each set of 6 blocks will have 4 good blocks and 2 bad blocks. Technically, this is only 33.3% corruption. And for a RAID-5 of 10 drives, we would only get 10% corruption. That doesn’t sound so bad! But the bitter truth is that the customer wants healthy files, not sectors. And there will be very few files that are not damaged.
Let’s calculate for our example: if the block size is 64 KB, then 6 blocks are only 384 KB. Almost all common files such as archives, photos and documents are much larger than this size, i.e. they are almost guaranteed to be corrupted.
Is it possible to recover data without a failed drive?
Let’s see if there is a way to recover data without the old failed drive C.
We have found that in a configuration with members A, B and NewC, the data will be corrupted. This is not a viable option.
What if we try to recover the data of drive C from redundancy? Unfortunately, during reinitialization, the XOR blocks were recalculated and now A xor B = NewC. This means that if we assemble an array of A, B, and dummy (a placeholder for the missing drive), we will get exactly the same data as if we had assembled an array of A, B, and NewC.
Conclusion: it is impossible to fully recover data without the original C drive.
Is it possible to recover data using an image of a damaged drive?
Let’s assume that the damage to drive C was not fatal, and we managed to make a copy of the data using PC-3000. Will this copy help us?
Data on drives A, B, and image C
Only XOR blocks on A and B are corrupted
The case in which the re-initialization has been performed with the same configuration is quite simple:
- the data blocks on A and B are valid
- the XOR blocks on A and B are invalid
- the data blocks and XOR on C are valid
There is one additional factor to consider — the array can store metadata about the configuration. The metadata on drives A, B may conflict with the metadata on drive C.
So we have all the data blocks, but there are XOR and metadata problems.
Data from RAID assembled from drives A, B and image of C.
All data blocks are correct — it is possible to get the RAID data without any damage.
With PC-3000 RAID systems, we can easily get the correct array data and keep all user files intact. Neither metadata conflict nor XOR issues will stop us.
Can the data be accessed with the original controller? Unfortunately, we cannot guarantee this, nor can we guarantee that it is safe for the data. Due to a conflict in the metadata, the controller may only build an array of drives A, B, and dummy (we discussed above that this configuration is not correct), or it may offer to rebuild the array: A xor B -> C, and this procedure will corrupt the data. Be sure to make copies of all drives if you want to use the original controller!
Conclusion: In PC-3000 RAID systems, we can assemble a RAID with intact user data. Using the original controller to access the data may be dangerous.
Conclusion
In this article, we have looked at RAID-5, the most popular level of RAID. Unfortunately, this level is also subject to the improper rebuild problem — a situation where initialization is started instead of a proper rebuild. This mistake is often made by clients when they try to regain access to data on their own.
In this article, we limited ourselves to the case where the new configuration matches the old configuration. We have used this example to explore:
- RAID-5 is a level that stores special XOR blocks in each row, which makes it resilient against the failure of any one member.
- During RAID-5 initialization, XOR blocks are usually computed and written. That is, each member of the array is partially updated.
- During a (proper) rebuild, the data of the failed member is recovered and written to a new drive. The data is recovered using XOR blocks. Thus, during a proper rebuild, only the new drive is updated.
- During an improper rebuild, the old members are partially updated with incorrect data due to the XOR recalculation.
- If a faulty member is permanently damaged, it will not be possible to fully recover the RAID data. All medium and large files will contain errors.
- If it is possible to make a copy of the data of the damaged member, the RAID data can be recovered using PC-3000 RAID systems.
- It may be dangerous to use the original controller for data recovery.
We have not considered cases where the new configuration is different from the old one. The damage there is more complex, but data recovery is still possible in many cases. We will look at some examples in the following articles.
Loading...