CMC were real-time services design, not fault-tolerant, hopefully, final cut. Okay, hello. We last time talked about a straightforward design for synchronizing a real-time embedded system to an external physical process, namely a ticking clock, and we saw why it doesn't work. It was the obvious solution. It's a best-effort solution. It had problems with I/O coupling. It was not rate-monotonic other than it basically had one service that could be given an entire CPU core. It might work, but the I/O coupling would defeat it. It would very likely have glitches where it observed the ticking process, it might skip a tick because the I/O blocking, which could be relatively unbounded number of issues with it. It was nice because it was simple and straightforward. A single service, just acquiring frames and store them and just provide a trace to syslog. That would work for post-processing. Someone could post-process and turn that into a perfect glitch list set of observations perhaps, maybe even not if the I/O really caused significant blocking, but it wouldn't work for a real-time system, for example, that could be turned into a flight control system where in our scenario, you had to fly from San Francisco to Hong Kong, glitch list the whole way or the plane falls out of the sky. It's that simple. It won't work. Let's move on and try again. Now, we'll present a better design that's also relatively straightforward and alluringly simple. It has problems as well. Many of you have maybe already thought of this. Take the previous design, get rid of the I/O coupling, so let's do that, and come up with a global synchronization scheme for the clock domains. Add something like a GPS receiver to the microprocessor embedded system design and we can get GPS synchronized UTC and we'll be home-free. Let's see what's wrong with that, let's draw out that solution and see where we can poke holes in it. It basically works, but it's just not fault tolerant. There are a number of issues that I've listed over here on the left, but let's draw out the solution and see how we can do. One of the things that we should do is we should create a ring buffer. You probably remember ring buffer from other courses you've had, but we'll draw one and remind you what it is. It's this buffer that's a wrap-around buffer. It's another name that people use for it, where we've got a number of frames and we take frame acquisition, best effort. We'll assume it's still best effort, service. We won't go straight to real-time yet. We have frame acq, and it can be best efforts still at this point. But what we're going to add is some frame write-back, let's call it WB for write-back. It's going to pull frames out of this ring buffer so that we're at least going to decouple the I/O. This is in memory, this is RAM-based. Therefore, there's not a significant blocking I/O through a driver interface. There's some latency, of course, accessing memory, but far less than doing it right to something like a NAND Flash file system. We have a frame write-back service which is independent and we then write these into a storage log, so we'll have an observing log, the observing log that we had before, of frames. We have solved one problem. We've gotten rid of I/O coupling, and this write-back will definitely be best effort. It's okay, could just try to pull frames out of here and write them back as best it can. Now, you're putting frames in and you're taking frames out, so there's obviously logic you had to take care of to make sure that you're only pulling good frames out of here and head doesn't meet tail and all those things. But those are standard functional correctness challenges that we would have in any software like this. We now have the same camera, and let's say it still runs at 30 fps. That's plenty fast enough, so 30 fps for the one hertz observing goal that we have. We watch the same clock, we have the same clock over here, and we'll put the 10 and the nine there, and we'll assume that it's at nine and it's going to 10. We're trying to watch this clock for a long period of time and have zero glitches in our observing. What we'd like to do is only put true unique tick positions of the clock into the observing log. We don't want to just fill the log with as many frames as we can best ever, but let's not worry about that for now. Let's assume that we could go through that later. Well, this is a better design and it basically works, we'll see that it still has issues. What we're going to add to this that really helps what, we'll keep the same viewer if you want a viewer. So if you want to see what's going on for debugging purposes, you can still use something like open CV and have an IMshow, so you can see what you're acquiring from the camera as you acquire it. By the way, this might slow things down. So try it, anything you do to debug can sometimes slow down a real-time system. One of the famous terms for that is calling that a Heisenbug. If you look at it, the problem changes or goes away sometimes even and if you don't look at it, then it's different. So you don't get to look at. Be careful with things like IMshow. Now what we're going to do is we're going to say, we have GPS time. GPS time is basically an atomic clock, onboard the spacecraft. You can get an atomic clock on the ground, etc. But the good news is NIST has an atomic clock and you can just subscribe to their atomic clock. You can get it over the network. You can get it over RF. I happened to be wearing a UTC synchronized watch, it gets the RF signal from NIST. We can get time from NIST. NIST provides UTC for example and you can get that over what's called NTP, Network Time Protocol over a wired Ethernet. I know for sure maybe over wireless too and you can safely assume for the one-hertz requirement that, excuse me, while I looked at the diagram here. That these are NIST and GPS are going to be synchronized. Certainly down to a second. You could even get a GPS clock if you wanted to. What I'm going to do here is show a little wire or conceptual wire or type thing where you could have GPS time coming into this clock, feeding this clock. So you could get a GPS clock or you could have NIST UTC feeding that. It turns out that you could do the same thing over here for your frame acquisition. We'll put the same conceptual wire or selector here. You could use either time standard to decide when you're going to read a frame from the camera. We're going to either take GPS time that's feeding our other clock or UTC time from NIST over NTP or something like that for FRAME Mac. It really does kind of solved the problem. We've got this clock over here synchronized from NIST GPS and we've got our FRAME Mac synchronized from GPS or NIST, problem-solved. This is still best effort, but it's best effort and it's triggered by this globally coordinated or globally synchronized clock source. A global clock sources awesome. Definitely use it if you haven't. What's the problem? Well, one of the things is, for example, I do research on GPS spoofing GPS jamming, GPS denied. So you might lose these signals. This is going to be over a network, over RF, wireless or wired network, RF receipt like my watch or GPS. If you aren't receiving this globally synchronized time, guess what? You've got a glitch and your plane falls out the sky, that simple. So it's not very fault-tolerant. This is actually a problem with a lot of modern systems, they are overly reliant on global clock sources like GPS. I consider it unacceptable. It works, but it's not fault-tolerant. It's still best-effort essentially, it's just triggered by a globally synchronized clock and then this best-effort service acquires a frame, puts it in RAM, and this write-back decouples the IO. We've definitely solve some big problems here. The problem is that any signal failure means we've got a glitch. The clock distribution also has to be over a low latency network. So not only do you have to have RF transmission and you can't lose the signal, it has to be low latency. You could use a backup clock source like you'd use UTC and GPS, certainly with GNSS, Global Navigation Satellite Systems. You can receive the US GPS signal, the Russian one, the Chinese one, etc. There are ways to make this more fault-tolerant. But in the end, if you lose that globally synchronized clock, you fall out of the sky and I would also add. You should add in the tracing, and we can fix that pretty easily. We can have another trace here to CIS log and fix that. Those are the problems with that one so better could basically work, but not fault-tolerant is the final analysis. Let's see if we can do any better than this. We'll try again and we'll have another segment on the next potential solution. Thank you very much.
