Radio in an IP environment
The DMR for Electric Utilities Roundtable was a moderated open discussion. Several themes reoccurred through the day. This transcript pulls together the phases of the discussion that centered on the theme, "Radio in an IP Environment."
Moderator: We will be getting into the security aspects. But, you know, the interesting point here is that this seems to be another juncture at which you want to say, well, what is our strategy for managing this whole thing? I mean, all of the IP connectivity allows for the possibility of things managing themselves more towards the edge of the network and being more dynamic and talk. You just plug stuff in. It says, "hi, I'm such and such a device with this IP address, and I'm part of this group here, and we will manage ourselves". But on the other hand, as a utility manager, you've lost a bit of control. But if you insist on centralizing the control, you run out of bandwidth. And if you add security, that puts more load on the bandwidth. So where does the utility go? What is the feeling? Are we still going to want centralized management? If so, what or are we going to go only distributed.
Member: You need to be a little careful, because it is not an either/or.
Member: And it is not an "everything has to talk to everything", and you have to break perimeters. It is a careful set of deliberate steps about what you are going to do and what you are not going to do. So, for instance, there is a big project on home area networks where, yes, it is a very decentralized sort of a thing. But nothing is allowed to join if the IP address and the MAC address aren't on a verified list from the manufacturer. And so you are just not allowed in if a central location doesn't already have from the manufacturer the appropriate MAC address information, and so on and so forth, and that has to be verified. Once that is verified, you are allowed to be a part of the network, but the firewall and meter says, nothing from inside of that home area network goes further up the utility communications network. So I will broadcast into the home, and I will take actions to say, "yes, you heard it", but in terms of other information coming back, "no, sir, none of that information comes back". And by carefully thinking through the layers and the way things work, you can have a combination of both centralized and decentralized, which is a lot safer to operate and gives you enough control to feel comfortable, but doesn't force you to have, you know, gigawatts of bandwidth, (pardon, wrong term, but you get the idea).
Member: I have seen in just in Brazil, the communication. Wherever there is sufficient coverage of, let's say, cellular system or cable system, utilities would try to use them for some of their communication needs. For instance, records list, when you have to close, again, the switch, they can use a device that is powered by our cellular system. They use some cable system, and some of the utilities have their own cable. They are service providers. They are cable service providers. For instance, one in (inaudible) they do it. It is a part of the Smart Grid, but they can't really trust the service provider, for instance, because it will go out much quicker than their own system. And I have seen those things happen consistently.
Member: If you have a huge customer base of 200,000 cable TV viewers and you have a utility which maybe has 50 end points, you are going to focus possibility, prioritize those.
Member: Of course. Same for cellular. Because they have one meter subscriber, and you have only so many technicians on the job - they provide the service to the one million subscribers and not to the places where utility wants to have the coverage.
Moderator: So is this where radio steps in?
Member: Of course.
Moderator: It provides for the critical communications, a utility owned communications technology, which isn't going to suffer from outages at all. Sorry, I will get Klaus and then Dennis.
Member: I think the way to look at this, perhaps, in a better light is through use cases. Define the application by the communications parameters that are needed for the application. Coverage is one thing. Latency is another one. Reliability is another one. And oftentimes, the bandwidth needed to carry the signals is ancillary. I mean, we have critical communications in grid monitoring that consume a very small amount of bytes, but they are very critical. In other areas, you can use a commercial provider for a use case that doesn't require such low latency. For example, Consumers Energy just went with cellular network for their entire meter reading system. Essentially, went with a vendor called Smart (unintelligible.) They felt it was a better solution for them rather than pay the upfront capex cost of building a huge network in an area where they feel cellular has a very robust coverage. They elected to increase opex cost for their metering network and just pay every year instead of taking a huge hit.
Member: You touched on reliability a little while ago. Right now, we were operating on two axis in the meter reading system on Delco, regular full wire circuits. And it got to the point when we had one go out, it may take two weeks to get it repaired. So, in light of what was going on, we switched all of the four wires, upgraded all of the substation units to Raven cellular modems. Now we are reading all of that through Raven.
Eventually, the DOE money is going to replace those with their own network. We are big on being able to control the network, the infrastructure. If it is a problem, we know it is our problem, and we get to fix it. We don't have to wait on anybody else to fix it. And that has worked very well. Our outages, we pride ourselves on being able to get a communication outage repaired in two hours or less from the time we know about it. So that is what our goal is, is the reliability. And part of that is [to use] layers. You know, we have a corporate network, and we have a set of VLANs that the Smart Grid will be off to the side. You won't be able to get into the corporate network, because it will be in this the VLAN network, which will be firewalled as it goes out.
I'm absolutely in agreement with Kathleen. We don't have enough bandwidth in spectrum available to the industry. I do appreciate the fact congress finally said, hey, you can have some of D block over the FCC's objections. And Klaus [Bender], I want to thank the UTC and all the work you guys did to at least get us a toe hold in that space.
Member: It is much appreciated.
Moderator: Spectrum is absolutely critical for utilities right now. I mean, this is something that public safety has known about for some time, you know. They're mission critical. They use radio because they can rely on it. It doesn't matter if there are other options there. This is the core of their service for utilities. If I hear you correctly, Doug, maintenance of what you've got right now is absolutely critical in the communication to make sure that maintenance is done as expeditiously as possible is essential. So this is where, in spite of all of the IP options, radio sits. The question is, is DMR as a digital radio, that can work in an IP environment across IP networks. Is this a good thing for utilities or not? And, what can be done to make the best use of the scarce spectrum that is out there? The FCC certainly would have liked, I'm sure, to have managed that spectrum for utilities a lot better, but, you know, it is what it is.
Member: I don't know how it works here, but in Brazil, you have to move into digital for two reasons. Because Brazil FCC requires them to narrowband from 25kHz down to 12.5kHz. And second, for some reason, it is only known to the Anatel - the Brazilian FCC - they declared that we cannot type approve analog radios from a certain date. Then every single utility in Brazil has to move into digital. And how many digital options there are, there are we already there is P25, there is DMR - two flavors of DMR - and there is Tetra. Tetra is something you don't use in - the United States, but it is available in Brazil. Then, each utility will have to synch it. They cannot do anything else. There is another problem that I don't know if you have here. Yesterday we went into the desert and our cells were still working kilometers from [any population]. But in Brazil, that is not true. You go into into remote areas; you have no cellular coverage, not at all. Then, that is the utilities have to put their own networks. We are just now finishing a commissioning of our transmission utility with 80 towers over some it is thousands and thousands of kilometers of digital network and transmission lines in places where there is no coverage at all for the utility. They need their own communications network, because they have to they are fined by government if something goes wrong, and it doesn't matter if it was a hurricane or if it was vandalism. It happens. They have to serve their market. And that market is much more important. So they need to have communication.
Member: Well, we might have had communications where we were at last night, but there are a lot of places in the United States that we don't have cell phone coverage.
Member: In populous areas, too.
Member: We weren't that far out of town. There are a lot more remote places than we were yesterday.
Member: Small number.
Member: I can't get cell coverage walking down the hallway at the hotel, either.
Member: It is a small number, but anyhow, I'm absolutely sure that you certainly have places, because where the the cellular people go to places where there are people, but your transmission lines will go through, through…
Moderator: …Places where there are no people.
Member: So I have a couple of comments about IP radio systems. So we operate an analog SmartZone Motorola system right now, and we are upgrading to a P25 core, which will be IP-based. And I have, you know, a couple of good things about it being IP-based. One, with a backup control center, we will be able to route automatically if we lose our main control center, we will be able to have our radio system. We will be able to recover your radio system at the backup control center basically automatically. So there is a benefit. However, we had an incident at my company last Saturday when our network, our corporate network, our SCADA network all crashed. They are supposedly not connected. I don't know a whole lot about the network side of it, because I'm not a network engineer. I'm a telecom engineer. I have a lot of questions from my operations people, because our mobile radio system is the last resort for communications. Everything goes down, SCADA communication goes down, but mobile radio has to work, has to be there. We haven't found the root cause of what happened, but they are concerned that if this were to take down our EMS system, took down our corporate network, what would happen to our mobile radio system, also IP, which is supposedly separate? Are they [the mobile radio network and the other networks] still somewhat connected? You know, and so there is a lot of concern about that, and I think that that is going to be I think that is going to be a case for a lot of people when they get into having IP-based radio systems.
Member: We have implemented cellular IP systems, for utilities, for public safety, and for trains. And you have no mission critical no more mission critical than train. Every single communication is going over the radio. And in IP, IP simply is the same as wire, because it is only internal protocol. It is not really connected to the external -networks.
Member: But they are somewhat
Member: They are somewhat. I think it depends how your network is designed. And like I said, our EMS system is supposedly not connected to our corporate network either, and our EMS system also went down. Where those connection points, you know, and how are they connected? I don't know, because I'm not a network engineer. But at some point, they are connected, and when you get instability in your IP network, what is I mean, the question really is by our operation center, what is that going to do to our radio network? And I think right now we I can't answer that. I'm the radio engineer. I'm the one putting it in. I can't answer that. I have no idea what it will do and where those connection points are, because I rely on my network engineer for the design of it.
Member: Listen, but I have no single incident that really something happened because it is an IP system.
Member: But possibly you have a network where they are where they are completely separate and there is absolutely no tie. I think there is a way to do that, as well, so that it truly is its own network that has nothing tied to it.
Member: It has its own network. No -
Member: So it is not tied at all to a corporate network or -
Member: Let's say that your system is connected by IP switches because you want it, to a telephone system.
Moderator: One at a time, please.
Member: This telephone system is completely separated. The telephone poles can fall down. It will not interfere with your radios.
Member: I think it comes back to system design and -
Moderator: I think Doug has got a question.
Member: It is not so much a question, but it is a comment. You can put a system in and it can be completely separate, and you can rely on people to keep it completely separate. But over time, as you do corporate network restructuring and other things, people begin to justify the cost of the new core equipment by being able to do converged networks. And you will hear Cisco and Alcatel, Lewison, and a number of other people talk about network convergence of equipment. And so inadvertently over time - not by intention, not by design, but inadvertently over time - when things are in place for 10, 15, 20, 30 years, you may end up with things that have been converged that you have no clue have been converged, and there was never any intention. And so when you make it possible to converge things, over time, somebody who doesn't understand why they are not converged, may end up doing that and untangling that convergence after it has happened inside a large enterprise is expensive, time consuming, and in many cases, there is nobody left who knows how it got there in the first place.
Moderator: Jumping ahead to one topic, does anyone use MPLS to manage their network?
Moderator: So, in part, you know, the purpose of MPLS was to provision a network design that allowed for an automated backup, new fallback routes, and so forth. Sorry, Kelly.
Member: I was just going to kind of moderate a little here, in my opinion.
Member: And that could be, Kathleen is saying something that is very important. You could build your two way radio system, a DMR trunked radio system, let's say, and you come from a dispatcher at a console through a special cable to a special router that goes to the network of the DMR system and gets to the base system through the switching core of the DMR system out to the air, out to the vehicles. That is completely separate from the PC based printers and other things.But Doug is saying what will happen is somebody will look in the closet and say, I have two routers here. Let's just collapse them into one and get rid of 50 percent of my network management costs without realizing Router A is the DMR router and Router B is the rest-of-the-building router. And all of a sudden you have got the crossover. And if there is a problem in the enterprise network, it could block your dispatchers from talking out to the radio system. Whereas, if it is on TDM or analog channels - like the old conventional systems - when they press the key, they are going out to wires, copper, out to wherever and not through the router that is currently handling the rest of the building.
Member: Also to Kathleen's point, if you build let's say you build a Tait network, like we built, and we have a support agreement with Tait. Somewhere along the line, Tait needs to get into that thing. So how do you get there? It is through the corporate wide-area network. There is that connection to provide a remote network management. So as harmless as that may seem, there a connection right there, even though, more than likely, you will need your vendor to get in.
Member: Even on initial design in our particular system, we have a network engineer who is comfortable with Cisco equipment. I want to use these routers, not the routers that the manufacturer is recommending. So there is even a battle, you know, on the initial design of the system. And what ramifications does this have over, you know, on your network stability, as well?
Member: So override your TI engineer.
Member: But me, as an RF engineer, I don't understand the network well enough to be able to say that this is going to cause a problem, this is going to cause a problem. There is a reason we have a network engineer, and I'm not that person.
Member: I understand.
Member: Sometimes you need to articulate what the requirements are to the network engineer, because assuming that they know [enough] is an incorrect assumption. But going back to more basic things, your claim that your network is IP-based is somewhat incorrect. It is not IP over the air. It is doing a translation. The over-air protocol is not IP. So a failure in your IP network does not inherently affect your radio network. Assuming, to your point, that the network's design corrected.
Member: If you have a trunked radio system, yes, it is.
Member: No, it is not. It is not IP over the air.
Member: No, no. But you are not able to communicate using your radio system other than from your base station, this group within that particular area -
Member: You may not have wide area networking, to your point.
Member: But when you design a trunked radio system, you are basing your design on being able to have wide area and for your dispatch to be able to have communication, which if your IP core goes down, you don't have - you don't have communication from your dispatch out to your field anymore.
Member: You are using the term IP core, and if you put all of your eggs in one basket, you are asking for trouble to begin with, all right. So you do have to segregate those portions of things that are mission critical. And to my prior point, if you are relying on wide area coverage, as would be the case with many utilities because of the extreme geographic areas they serve, then this is where you need multiple connectivity backup. You can put site comm backup on so you can use your low cost provider, even your own infrastructure, whatever you have, but have extraterrestrial backup to your systems. And if they are critical to you and you don't do that, shame on you.
Member: But you can't give a satellite radio to every one of your crews.
Member: No, no. You are talking two different things. Your radio system is not IP-based. Your entire core can go down and, other than perhaps the connectivity between the sites that are in your network, the system will continue to function. The only thing you lose is the wide area connectivity portion of this. And you back that up with multiple ways - not, you know, with copper, which I agree with Doug is too often converged someplace, and, you know, you are fooling yourself.
Member: I'm not even talking the communication to the remote sites. I'm talking the core equipment that sits at the prime site.
Member: By prime site, are you talking about your dispatch center?
Member: Yes, not the dispatch equipment. It is all the networking equipment that makes up the trunk system.
Member: But if you have a control system.
Member: That is all IP-based. If you have a control system to your network, your radio network that sits on the LAN that your comm consoles are on, the rest of your core IP network could go down and it still would be functional.
Moderator: Dennis, I have a question here, and then one from Jonathan.
Member: When we designed air trunk system, when I worked at North Anem, which is a nuclear power station redundancy was the key. Redundancy upon redundancy. This if this goes down, then it goes to here. And if that went down, then somewhere else. What I did there, and it was a Motorola SmartNet system, and what I have done for the Tait system, is that I put in RF base stations. And in every dispatch center, I do not rely on any wires on the backup. It is all RF. It is just a desktop unit. It is a mobile unit attached to a 12 volt power supply. You can talk from there to the network just like you would a truck. Because if you go on the assumption, that if you lost all of your IP connectivity, the network was down, then that also means that all of our outage management, all of that is going to be down to the districts also, because all of that resides in our corporate office. So you are going to be opening up the districts for outage control anyway. You go back to the old maps. You go back to cards, [manually] keeping track of everything that you are doing. You put a guy on the radio, and he controls the crews from that one radio. So, yeah, it gets very simplistic, but you've got communications in the field.
Moderator: I think the use cases are absolutely everything. If you are talking about whose communication when the network goes down do you really want to have staying up? And then, what is the backup network for them? And I know that even if you are not talking about maintenance networks, that you are just talking about, say, SCADA networks, some of the utilities that I visited in the Middle East actually backed up these: the radio was the bottom line backup, but they had copper wire. They also had fiber, and all of these were fallback positions. So when we put a radio system in, it was really just one of the backup systems, or backup networks. They were all completely independent of each other.But as Doug rightly points out now, you know, over time, for various reasons, something to do with ignorance, but something to do with pressure from management, the inability to support these develops because the service just isn't there anymore. So these backup communications paths and networks start converging. And, again, when you have separate networks, you've got redundancy, but you've also got a maintenance issue, because these are separate, and they will start to lose synchronization with each other. And you have to maintain key technicians and skills to maintain separate technologies that are going to be maybe at even different generations.
Member: I think one of the questions that I would have, because we don't do mobile data on our system yet, [is about DMR and AVL]. We are contemplating doing AVL, and I know when we talk about doing work order dispatch for a distribution cooperative, and stuff like that, and the middleware, which is pretty much, I believe, the same thing you are talking about, strips off the sensitive data that changes. I think there is an unknown as to what that cost is. And do I have to have different middleware for different applications. I have 28 distribution cooperatives that I would potentially be supporting, 28 different work order dispatch systems. Do each one of them have to have separate types of middleware?And so [the problem is] trying to figure out how that works, and we are at the very beginning of even looking at the capabilities of putting data onto a radio system. But I think the costs are unknown. And how does that look? And I also hear about you can transfer things from WiFi when you are in your office, and the middleware will route it to your private network when you get into your private network. But, you know, until you start doing a design, honestly I have no idea. Is it $3 million? It is $30,000? I really have not even a ballpark as to what it would be. So that is kind of where I'm coming from as a challenge right now.
Member: I think you will find that it will vary heavily depending on what you are starting from. You know, folks who are starting from older applications that are used to having fat clients where a large percentage of the data lives on the actual client, the middleware is pretty simple and pretty cheap and those sorts of things. And those people who have gone to newer thin clients where you are using a web app, you find out that all of a sudden there is a lot more that you have to deal with and you've got to generate a fat client out on the laptop or whatever device is out in the field. And that becomes a much more expensive way of doing things.And so for some utilities, where they haven't upgraded a lot of their applications, they have a lower cost, and for other people, who went forward and put newer generations of applications in, they may have a higher cost to deal with. I think one of the best places to go and see how it works is at Consumers Energy, because Wayne Longcore (phonetic), who was there for years and spent a significant amount of time figuring out how to get all of his applications running in a fat client world with very, very narrow pipes on an analog radio system. And they've done a really, really wonderful job with it. I know, Kelly, you have done some mobile data at Oncor. I don't know the extent, because I haven't been down there in three or four years, but maybe you can talk to some of it, too.
Member: Would I need - if I have 28 different companies that have 28 different I wouldn't say applications, because 28 different companies that might have multiple applications, I could potentially have hundreds of different applications. Do I have to deal with all of them, or is it one piece of middleware that handles all of that, or do you have to have different pieces of middleware for every application in every type of system?
Member: I don't think you have to have different [middleware] for every, but you may need different adapters for each so that you get the data in in the right form and sort out what you need to send and [what you] don't need to send.
Member: I would think in your position that you would be able to drive the ship, because the distribution cooperatives generally listen to you.
Member: But they all have different billing systems, different ways that they communicate with their customers, different forms. They are not the same. So that gets to be very costly for different regional cooperatives who don't have a lot of money.
Member: One of the tools I've routinely used is something called Click. And Click offers a tremendous number of different pieces already and a lot of adaptability in what it does. I'm not going to tell you it will solve your problem, because I don't know your problem well enough.
Member: I don't either.
Member: But at least that is a starting point that you may want to look at and look at a couple of case studies from Click. And as I said, I think Kelly has done some of this work, and I think the guys at Consumers have done a lot of this work, and they can probably provide you with reasonable lessons learned.
Member: One of the things that that is difficult to talk about, is the fact that there are two forms of DMR: dPMR, DMR, and the various manufacturers have taken some poetic license with the standard as to what their version of what DMR or dPMR is. What one of the things I'm seeing thus far is much better treatment of mobile data to address the problems you are talking about. You can in one platform, I know you can run anything that is 80 kilobytes or less over the control channel and never affect the voice channel. If it exceeds 80 kilobytes, then the payload can trunk off to a data channel or trunk off to a conventional channel. So you are in much better control and can prevent the collisions that you have run into in prior systems.
To the other point about middleware, as we call it, and Kathy's concern about applications, there are several things. Even in the analog world, I have seen some smarter data manufacturers that have written the core program without (inaudible) and allowed this core program to address virtually any third party application that needs to be brought into the core program.
And I would assume that -(I think Kelly brought up the concept) a lot of the existing infrastructure is very chatty, to put it lightly. Unfortunately, most code writers - and I've worked with the manufacturing sector for a number of years, and I found it to be true - think they always have an unlimited pipe, and they do not write applications that are bandwidth-conservative at all. So certainly, middleware is a great way to address that, particularly if you've got legacy systems out there where you could take and filter that down, do exception reporting and cut down the chatter, to make things a little more realistic.
But I always like to go back and recite a real life case because, long term, if you are going to get best mileage out of the narrowband platform that we have, it involves code writers that understand the medium they have to work with. The best example I know of from past is the state of North Carolina built a 9.6 kbs statewide network for public safety. They wanted to be able to do facial recognition. Law enforcement officer, or state trooper, stops somebody, and they want to know if this is really the person they are claiming to be. They say, well, we have the licensed database with photographs, so we will just take and convert that to digital image. And digital image came out to be something like 128kb, whatever, and over a 9.6 kbps network that wasn't happening. They went back, and they said, Jeez, all we really need is 'this'. And they got it down to a 1kb payload.
So it goes to show you that, as an industry, you set standards for the applications and tell them what it is that you really need. If you just do it at too high a level, then you are going to get bandwidth intensive applications unfortunately. Because the people that write these applications don't live in our world. They live in these ivy tower little palaces. They have no idea what is going on out in the real world and what your requirements are. If you verbalize that and bring that to them and say it is unacceptable to deliver non bandwidth conservative applications, then you will get their attention, and you will get what you need, and you will better be able to use the spectrum and stuff and the resources you have.
Moderator: Vernon, and then Rob…
Member: When he talked about DMR, and I have a real concern that the bandwidth we have available to work with, I mean, you can do all kinds of compression and middleware, but one experience I've never had is to have anyone come to me and say, I've written this application and I don't need any more bandwidth. In fact, I could use less. I've never heard that. So what happens is: we have an application that allows people to do service orders, and they come back and say, well, that is great, but today we are updating maps in their PCs in the trucks on all the WiFi and at the service center when they drive in. So they've got the service order. Now they want the new map. So to my mind, the questions for DMR are: for how long it is going to satisfy us, and how quickly can it be rolled out? Because if it is three to four years before we can deploy DMR, the bandwidths are going to be way past it.
Member: You have to remember that when we talk about some of these applications - at least in the business radio world - we are often not operating on an exclusive channel. So there is duty cycle involved, as well. Which, again, supports the need for spectrum, but also really enforces the need for these applications to be as efficient as possible, especially when systems are operating in a shared environment where you are not allowed to transmit constant data streams and where you have to be considerate of other users who may not be on DMR. They may be using analog. They may have a variety of things.
Moderator: Of course, that is an opportunity for applications development. If developers have to work, for utilities, they basically know what the target platforms should be, and they can choose to be middleware providers (at the very least) or native application builders (at best) and so tailor their development to make best use of radio spectrum.
Member: Or if there are ways to aggregate disparate channels (so if you have a frequency here and a frequency here, you have a way of putting them together) then that you have more data capabilities. Because finding channels near each other is pretty much impossible.
Member: The four conversations I just heard lead into what I was just going to comment on actually. We started with number one, future of the digital connected utility. Then the conversation turned into the present needs of the utility as the utility is today. We talked a lot about communicating with the mobile work force. Communicating with the mobile work forces the way today's mobile work force operates. But tomorrow's mobile workforce is going to operate differently. You are going to have video imaging on site. If someone does go down, it is not going to be just that warning button. It is going to be, you know, instant communications to the hospital network who have their own safety network in place, visual imaging to the doctor who can make an off site call as to how to immediately treat that person. So in the utility of the future, the workforce of the future is going to be much more connected and those requirements are going to be bandwidth-intensive. There will be some less bandwidth intensive needs out there, and DMR must have a place for those types of needs.I think about my home area network where we are using the customer broadband to backhaul and download our firmware. It is a 40- 60Mb download of firmware to upgrade that device. We can't do that over our radio network. We won't be able to do that over, I don't believe, a DMR type network. It has to be done to those devices almost real-time. I mean, you can't take all night to download one device, so it is something to be cognizant of.
Moderator: How big did you say that was?
Member: 40 to 60Mb package for a firmware download is what the vendors told us when we do the firmware.
Moderator: They should rewrite their firmware.
Member: Big applications. These are the applications. I'm not sure what they are using them for, but the deal we have with the device manufacturers and eventually they will be offering energy management applications to the customer. The point I'm making is: today's utility doesn't look anything like tomorrow's utility will, and tomorrow's utility, in my observation, is going to require a lot of bandwidth.
Member: I think you also have to be able to get that bandwidth in order to provide those applications. And if we don't have access to that bandwidth, you may not be able to have those applications.
Member: You won't have access to it. There are other people that will have that bandwidth right now, so it is the hybrid network that needs to be.
Member: If that is the key, it is sort of the classic use case. I can see a mobile worker who has a use case for which DMR is the perfect fit. That is mission critical voice.
Member: Maybe the AVL in a new IP base radio system can be carried as well without having the traffic issues that we have with the analog system trying to carry the AVL, (automatic vehicle location) data. So the DMR system's data carrying capacity on top of (or underneath) the voice capacity will need to be rationed to those applications or use cases that it fits well.
Moderator: Excellent point.
Member: But it is also mission critical data potentially, because you are going to have an infrastructure system behind it that will stand up under outages of all types and sizes, including blackouts.So, then, with the other use cases - like maps to the trucks - I think what we've finally bumped into is [the limits of] cellular connections. We are going to end up with a truck that will have two or more connections to it all the time whenever it is inside the coverage of the systems. Now if that is an LTE system or 3Gor whatever that [connectivity] will come based on where you are. And if you run out from under it, you will still have your voice to fall back on, but that is all you will have.Then, if you have a man-down button that is pushed, the most you will get is the man-down signal and the location where he or she is working.
Member: One of the things I worked on was the idea of putting in secure WiFi and, in substations with high gain antennas, and providing store and forward capability in those substations. So I could push maps. I could push other large chunks of data to the substation in the vicinity of where the truck was or was going to be. And literally, we found out that for most of that stuff, if the truck goes by the substation, stops at the stop sign, and makes the turn at the corner, they've picked up that [data] package in making that trip past the substation. So it isn't even a stop in many cases. And that is, you know, a potentially useful way to get blobs of data out to the truck in the middle of the day. It might cost them 10 minutes to drive over close enough to the substation in between jobs, but it may be enough to solve a number of the problems.
Moderator: As long as the van has got some sort of place to pick up the WiFi, store it, and, if necessary, do some local processing and then forward it on to the core network…
Member: And the reason we like this is it kept everything on the utility's own network and within the utility's own security. And so they didn't worry about other people getting involved in it.
Member: So I could see potentially some other uses for DMR that would be at fixed locations. We are a very rural environment. We have, you know, substations and areas that people don't go into very often. And with the higher power that you are allowed with those types of frequencies, I could see doing a down line automation. I think there is some automation things that are not bandwidth-intensive and that are fixed. You could use a DMR technology for these in addition to using it as a technology for mobile voice.
Moderator: If I could summarize some of the threads of this discussion here. It looks as though, apart from mission critical voice, you have a number of application functions that you want to have carried out, and you've got a number of technologies available, all of which a utility may use. It won't fix on just one. There won't be one technology solution that does everything. But what you want to do is both prioritize the applications and/or to partition them up, partly in terms of their importance, partly in terms of how much bandwidth they require, and in terms of how much latency they can tolerate. And then basically, you are going to farm these different applications out to the different technologies that a utility will make use of. And, of course, you will have those applications that require the greatest coverage and reliability and for which the utility will own all of the supporting infrastructure and the terminal devices at the edge. That system will probably be your radio network.
Member: And I don't think you will ever find two utilities that look the same from a communication standpoint.
Member: The point I would like to make is that if you forget about trying to do some of these high-bandwidth applications over a DMR, there is some really basic things that would be really, really useful that you can do [with radio]. For example, you know, all of the basic fleet, sub-fleet type of things you can do on a trunk network. We don't really take advantage of all of those things. And every one of our people carries a cell phone, and then they've got an Aircard on a computer that is in their vehicle. Well, if you can get rid of the cost of that cell phone that everyone is carrying along with them and you can give them more of an interface [then radio has a case]. Right now it is like, if someone wants to make a phone call, they have to go through a whole bunch of things. Somebody wants to call them, you have to dial, and you have to do an over-dial. And people just don't want to use [this interface] just because it is way too clumsy. If you can make [a radio system] more like a regular telephone system and interface it with voicemail, things like that, then I think that you could justify putting in a DMR system and taking the cell phone out since utility users don't need really all of the smart phone type things. That is not what they were hired to do. But they've all got cell phones, and we are paying a monthly bill on every one of those. If you could use that radio system for their voice communications, I think that [you could save a lot].
Moderator: I will just make a point here. Some of the utilities that, have looked at this issue, have asked themselves are we going to put our communications money into capital expenditure, or are we going to put it into operating expenditure? And some of the utilities that looked at this and balked at the capex required - who put data stuff through GPRS or the cell phone system, have discovered their opex bill has just gone gigantic.One of the advantages of owning your own radio network, once you paid for it, the rest of the cost is just the maintenance and upgrades and keeping it healthy.
Moderator: Some of these options are available in the inside the products themselves, as you've alluded, and some of them are built into how you manage your network. My understanding, if MPLS, for example, is married to an IP network, it can actually tag the different loads that are going across it and then decide how and when they should go across the network. So part of your whole network management strategy can be determined by what you want to do and how you want to distribute the communications load, which is going to vary from customer to customer and data type to data type, and application to application.
Moderator: I think you will also find in the work being done by the IETF around rural, that there is a lot of tagging and prioritization that is going to be built into lightweight IP that they are working on for narrow pipes. So it is probably something worth keeping an eye on if you are interested in prioritizing and routing and so on and so forth. Even storing and forwarding traffic that isn't critical.
Member: Just so I can frame this up for my own benefit. Are you stating that you can take a technology like this and use this as kind of a monitoring and decision-making technology which brings on and off the different networks you have based on their need?
Member: You were saying, you are driving now. You got the narrowband application, and then you have this big packet, but you are not going to get it until you drive by the substation. And so you can take a technology like this, and it is going to be that kind of decision layer out there. Interesting.
Member: Another form of middleware, but in this case, a mobile form of middleware, if you will.
Member: I can give you an example. From now, a big railway tried to pass through their narrowband system their critical control train control and train operation - train operation signals. And it didn't work, because those guys, it is not on the proper bandwidth, but (inaudible.) Then they changed it. They put critical data on this narrowband system , and the data that, let's say, was [non-critical], important but heavy - like several megabytes - the train picked up only at places when he stopped, so WiFi. And they work like that now. The radio network is their mission critical system, because they need it. They are required to: the train operator has to know exactly where everybody else is and what is being done and if he got the right of way or not.
Member: That decision is made at a different communications layer, right, so different protocol?
Moderator: As I said, MPLS and tagging can do some of that as part of the management system. But what this says is that each customer has to decide "how real-time is this information?" How real-time does it need to be? As Bob alluded to, "near real-time" may be good enough. So waiting a bit may be good enough. And also, how reliable must this application be? And it is these decisions: - tradeoffs between bandwidth required, what you can do with middleware in order to reduce bandwidth requirements, how real-time an application needs to be. You can fit this into your communication strategy, and then you can decide, what are you going to put into DMR? What are you going to leave outside of it?
Member: Now, if had you been to some of the wireless conferences I've been to, you would hear a different story from the different wireless providers. When the buildings come down, they had the only systems up and running. Glossy advertisements there.
Member: So one of the very common utility applications of which DMR is fully capable is to deliver switching orders. That could even be done over short text messaging today. But we all use it, and today probably most of us do it over voice. Is it is subject to transcription errors with consequences.
Member: It is done. We have seen it implemented. It is in Brazil in many places - over narrowband radio you get only the data that you need. It needs the address, the name of the guy, and the call of the possible problem. When the guy goes there, he will check [the reported problem] and then he will send back the code of the real problem and say I'm ready or not for the next job - things like that. This is something that takes only one byte, or several bytes. Is very low bandwidth usage and you can do it [easily]. We did it on analog, not on digital, but it is a very simple thing. Now they are trying to do something else, such as communicating a list of materials that are used or something like that. Once again, it is simply a matter of some simple code, , Code 3 or something like that. Very simple.
Member: That is a good example of a lot of information that could be done with more latency. It doesn't need to be done in real-time.
Moderator: If you have got very little spectrum, the resource issue is a bit like application development planning in computers used to be: you know, processor time and random access memory, the really expensive critical stuff. You want to strip out everything that doesn't absolutely require those critical resources and move this either to offline or slower resources where it can be used only when the opportunity or need arises (so more latency is tolerated) or you move it to another technology altogether.
Member: I think if you look at things, I mean, DMR, properly designed and deployed, is inherently mission critical voice. The interesting thing is, it has got extensive data capabilities. The value of data is that it is non real-time information, and it can tolerate latency. I mean, there isn't a data network out there that doesn't have some latency component into it, and enough [networks] that you couldn't, in many cases, them for voice, because you lose packets here and there or whatever. Or by the time it retransmits, it is not in real-time. But if you are trying to throw a cut-off switch or decide whether you are going to shoot somebody or not, [this level of performance] is not acceptable. So [in DMR] you've got what amounts to a foundational real-time system that also has data capabilities that you could use as you see fit, making some intelligent decisions about what you move over it and when you move over it. And more significantly to me, from a technocrat's point of view, you have options for how you move it over the network. That is something is downplayed, but is really significant. We've never had that before. Even in analog trunk systems, we never had that degree of flexibility as to, you know, I'm going to move this data this way.
Member: Not all data can tolerate latency. If you are doing SCADA, if you are doing controls that has to be as immediate as much of voice is.
Member: But you are missing my point. You've got [in radio] a real-time network. So [you have an option] if you decide that [there] is something you need to or want to move over that network, or if you [simply] want to have backup to your existing networks. Because, as a number of us have pointed out, the outsource networks that are commonly in use today are inherently flawed. And I can point [out examples]. I mean, in the Northeast, we don't want to talk about the outages we've had in recent history that have affected everybody.
Member: I was going to say, I would never put control over any of those types of networks. We would only do control over as something that a communication network that is as robust as our voice system, even though the voice system has to be there even if that particular SCADA communication system goes away.
Member: You just made my point. We are talking about that robust a system, and your ability to move both real-time data and non real-time data over it. So, too, these enhanced things that I was talking about: like do you want to see the picture of the guy trying to get into your substation? You can do that. Is that critical enough to be moved as real-time? No. All right. And you can make intelligent decisions about that, so you can tag stuff, scatter data that may need to be real-time as real time and move it as such.
Moderator: Maybe you are in violent agreement here.
Member: No, no. I think that he just said that data can have latency. I was just pointing out that not all data can have latency.
Moderator: This is true.
Member: His preamble was that all data can have latency, and I have to agree with Kathy. There are some critical applications where latency is death.
Moderator: Like EEE [power systems protection] relay.
Member: A deterministic path, and deterministic processing time, and we want to measure latency in nanoseconds rather than measuring latency in milliseconds and those situations.
Member: But I hear you saying, or what others have said, that people are converting their old RS232serial data SCADA systems over to IP. But just in doing that, you [have] inherited latency. It is no longer real-time at that point in time, because you use processing time to do the convergence.
Member: When you talk about nanoseconds or even milliseconds, you are not talking about digital. (Unintelligible)
Member: It is a deterministic time period. If I know there is a 400ms delay on the signal, so long as it is 400ms every time, I'm okay. But if it is 400ms this time and 1,000ms next time, then I'm not okay. And when we look at our SCADA networks, there is a distance involved, so there is a delay. There is nothing we can do about that. There is a physical distance involved, and the further away I am from the actual back end, the longer that delay is. It is the law of physics. I haven't figured out how to repeal those, yet. And then there is some instrument time at both ends, and there is also some A-to-D time on the other end. So it isn't perfectly real-time. It isn't instantaneous, but it is highly deterministic how long it takes for that data to get there, and how it is going to be processed. And that, in many cases, is the critical thing. I need to be able to repeat it millions of times exactly the same way.
Member: Well, there are certain applications just relaying that you would never put on a wireless network because you can't do it in time.
Member: They are not deterministic enough and not short-enough-time period. I completely agree with you, Kathy.
Member: That is a point I would make is that in those instances there would be inter-tie subs or protective relay. In those instances, the applications would be inter-tie subs or protective relay, and at those locations I already have a need for other services. I wouldn't be looking for this tool to do that.