Introduction
As the NVMe protocol sits atop the PCIe protocol, it is necessary for the analyzer to first capture clear PCIe traffic in order to capture proper NVMe traffic. If the analyzer is already capturing error free PCIe traffic then we can skip to the Recording Options portion of this document. We can see how many errors there are in the trace by clicking the Error Report icon.
If there are errors in the trace there are a few likely reasons.
1. Polarity Inversion
Each lane in the PCIe link can have positive or negative polarity inversion. This is set during link negotiation. If the analyzer’s polarity settings are opposite to that of the PCIe link then it will only see errors on the inverted lanes. In the analyzer recording options there is an option to have the analyzer auto-detect the lane polarity.
The analyzer must be on and connected to the interposer during link negotiation in order for this feature to work. You can also manually set the link polarity of the analyzer in the recording options.
2. Lane Reversal
In PCIe, TLPs and DLLPs are stripped across each lane in the link. The lanes can be ordered sequentially or reversed. The lane ordering is also set during link negotiation. If the analyzer lane ordering is opposite from the PCIe link it is analyzing, each link may be in a solid “green” state, but only errors and SKIPs will be captured in the affected direction. (SKIPs are not striped across lanes and will be captured correctly). This can easily be flipped in the analyzer by checking the “Reverse Lanes” check box.
3. Calibration
When a PCIe link is running at 8GT/s, the analyzer may need to be calibrated for the specific signaling characteristics of that link. Begin when the PCIe link is in a stable L0 state. Go to the “Probe Settings” tab in the Recording Options. Click the Auto Calibration button.
When the calibration is complete, click the Record button to apply the acquired settings. It is faster to only calibrate in the direction that is detecting errors (usually the Downstream direction). The option “Quick Calibration” is the preferred method of calibration and will provide the fastest calibration results. If the Quick Calibration option does not provide a clear trace, then we can do a more thorough calibration by unchecking this option.
Setting the correct recording options in the analyzer will help in getting a trace that has everything necessary for NVMe analysis
- Set the buffer size to a large amount. The size required to capture the entire sequence depends greatly on environment it is connected to. A 1 gig buffer is a good place to start.
- Set a trigger for Any TLP or any Config. Everything before this event is link training and is not necessary to the decode of NVMe. Setting a trigger on this event will prevent the analyzer's buffer from filling up to quickly with training events.
- Set the pre/post trigger slider to the left, so that 10 percent of the trace buffer is pre-trigger and 90 percent is post trigger. This reserves most of the trace buffer for the configuration cycle after the link training necessary for NVMe decoding.
- Set a filter for SKIP and UpdateFC. These events are not necessary for NVMe and will fill up much of the trace buffer if they are not filtered out.
We are now ready to capture the boot up trace. Begin with the PCIe link completely powered down. Click the record button, and then boot up the system we want to analyze. We will immediately see the first 10% (the same amount we allocated for pre-trigger in our recording options) of the buffer fill with training sequences. As we are not interested the training sequence at this time, we will continue waiting until the analyzer triggers on the first Config TLP. The total length of the configuration cycle may vary. It is important to capture the entire cycle. After the config cycle the analyzer will continue capturing NVMe queries and queue creation. Once the buffer fills up, or boot sequence is completed (we can click the Stop Recording button if we know the host is done creating all the queues) the resulting trace should have everything we need to begin analyzing NVMe.
After the trace is captured, the GUI will show a PCIe level view of the trace. Click the NVM button in the tool bar to start the NVMe decoding. When this is finished, NVMe level events should appear in the trace.
Pressing the NVM button will cause the software to fill out its “SSD Base Address Map”. You can view this map by going to “Tools” in the toolbar of the user interface.
This map contains all the information that the PCIe Protocol Suite will used to decode NVMe protocol. Within the trace there should be a Configuration Read of the Class Code of the device.
If this is an NVMe device then we should be able to see that the Class Code is equal to 0x010802. This shows that the device is an NVMe device and the MBAR, queue creation, and other information necessary for the decoding of NVMe will be associated with this device number.
If the Class Code is not equal to 0x010802 then this device cannot be NVMe. It may use another storage protocol such as AHCI, or it may not be a storage device at all.
The trace will also be filled with training sequences, configuration writes and reads, and other non-NVMe events. We can quickly move to the NVMe events we are looking for by using the analyzer’s search feature.
