Case A: The simplest one, debayer'ing is done at hardware level, and so the data coming from the camera is already in full color. It uses the most Firewire bandwidth, as well as hard disk space and bandwith, but this is the simplest one. Since debayer'ing is done by the camera, it's good for those who has slower CPU.
Case B: By choosing Y800 for the device codec, the hardware sends original data which is not yet debayer'ed, that saves some Firewire bandwidth, and then the debayer'ing is done at software level by IC capture (see the pull down menu), and since the data stream is already debayer'ed, you will also need RGB24 to store the information. It uses less Firewire bandwidth, but it uses the same hard drive bandwidth and space as Case (A). In that case, more CPU cycle is used for Debayer'ing, so if you got a relatively slow CPU, you will get less maximum fps.
Case C: We have chosen Y800 for the device as above, so the data coming from the camera is still not yet debayer'ed, and therefore, it uses less Firewire bandwidth. And Debayer'ing is also disabled in IC Capture, meaning that Debayer'ing is not done on-the-fly as well, and so it requires less CPU cycle. And since the data is not debayer'ed, we can use Y800 to store the data as well, that uses less hard disk space as well as bandwidth.
As you can see, this combination uses the least system resources, so if you got a slow CPU, a slow hard disk, this combination will give the highest possible fps. Debayer'ing must be done on Registax, however, but this is not a problem. Since Registax needs not to do things in real time, a slow CPU won't affect the quality or the frame rate (obviously).
Remember that you can only use Y800 as the storage codec in the last case, since for case A and B, the data is already debayer'ed and in color, and thus, if you use Y800, you lost information and it couldn't be recovered afterward.