The Future of Digital Radio Connected Utility
The DMR for Electric Utilities Roundtable was a moderated open discussion. Several themes reoccurred through the day. This transcript pulls together the phases of the discussion that centered on the theme, “The Future of the Digital Radio Connected Utility.”
Moderator: But I think, you know, the focus of this is really not on the technology as such, but how it fits with the changing utility industry. And one of the major points of discussion in how the utility industry is developing is Smart Grid. But, what is Smart Grid? You get a variety of different answers. So, anyone tell me what their understanding of Smart Grid for their utility is?
Member: This is a "has-been" term that has now been politicized, which is radioactive.
Member: I think you will find anybody who is talking about an inside of a utility today talking about grid modernization rather than Smart Grid.
Member: I heard a definition from the customer point of view. It is - it is not my definition, but someone said at a conference I was at - they said Smart Grid is when I can get the type of power when I want it at the cost I want it, the price I want to pay for it. To the, to the customer, that is the Smart Grid.
Moderator: And that is the end customer. Don't the utilities have an interest in the extent of their management and control of the utilities, how much they want to centralize, how much they want to decentralize. Isn't that part of it?
Member: Sure. The utilities are using grid modernization with advanced telecommunications capabilities in order to ensure their reliability of the grid, but at the same time, through demand reduction and other activities, reduce the requirement to build additional generating facilities. So it is one of the benefits of Smart Grid.
Member: So I would view Smart Grid as being on the distribution side. Because on the generation transmission side, the grid has been smart for decades. It is more of putting automation into the distribution network both for distribution automation and for AMI advanced metering infrastructure - and smart metering and the home area network. But I don't see really any Smart Grid being on the generation transmission side.
Member: So no PMUs.
Moderator: Well, I guess I don't consider PMUs to be Smart Grid. I mean, it is, but that is not really what I would include: that is another technology. I don't view that as being part of Smart Grid.
Member: And I think you will find a lot of people don't view distribution automation as being part of Smart Grid, but other people would. So I think everybody's definition of what it is a little different.
Member: I would agree with that.
Member: I think there you know, back in the days of mobile radio before you got your smart phones, you had to have that killer app. You know, as soon as you get that killer app. I was at a Smart Grid conference, and there are all these vendors showing all their neat little things, but the problem is: if your consumers don't want this stuff, until you have these apps that consumers want and that they are going to use, you know, it is like, are you really going to log in to find out, you know, the temperature of your you know, your swimming pool? Well, who is going to I mean, there is all these different “gee whiz” applications, but there has to be something consumers are willing to actually want to use. That is my take.
Member: There is a chicken-and-egg issue here, because smart metering is one area that may or may not be defined as Smart Grid. But intelligent metering is a tough case in a lot of ways to sell financially, because it is pretty cheap to read a meter manually once a month. But 15-minute interval data may offer advantages that we have yet to even all, you know, expand upon or determine. We just turned on in the last 30-45 days the ability for our intelligent meters to talk to our outage management system to create outage records without having to have the customer call. And it has taken us over a year of heavy integration, and we've been deploying smart meters since 2006, I think. So we've had a lot of “ahha” moments in this chicken-and-egg thing, says we wouldn't have really turned on this outage management reporting without having an intelligent meter in place, but we couldn't it also without not modern outage management system. So you really Smart Grid, to me, is the combination of intelligence in the field, intelligence at the head end, and communications to link the two.
Moderator: I mean, taking this to an extreme, it is basically the whole, you know, right from the meter or even beyond the meter into the management of the appliances in the house. If you really want to go to the science fiction, you know, right back to the generation, all of it is one connected network and all of it is managed. And, in fact, you mentioned about selling the killer app, selling to the customer. Well, in part, the customer is actually the utility itself, because it has got, you know, certain options for generation. It wants to make the most efficient use of its current generation capability, not to expand that beyond what it needs to. It wants the consumers to use their power better. It has got to manage outages, because often there are agreements that will penalize it for unavailability of service. So it is really, you know, there are two customers. There is the utility itself and the end customer who is paying for it.
Member: Towards the end customer, until you mention it, oftentimes, we find it to be a hard sell to the customer. Has the industry done much in reverse engineering, as I would sort of term it, as to what the end user really would view as compelling solutions that would drive their interest in this? And, you know, sort of looked at that as how you get to a true Smart Grid that is not just technology for technology sake, but technology for benefit sake?
Member: That is very important. We had, not as bad as in some areas of the U.S., we had a kind of initially a growing opposition to smart metering, to use that term, intelligent metering, because there were suspicions by the customers, the end use customers, the consumers that this was a plot to, one way or another, do them in negatively.
Member: It is not SkyNet.
Member: Yeah. And it is not a way to double their electric bill just because your old meter was not willing to cheat them, you know, whereas a smart meter just a little black box, and it can cheat the customer. We had to go so far as to put the smart meter in a dual socket with the old meter and show them over a period of months that they read the same, and in many cases, the smart meter actually read less than the old electro-mechanical meter. But in a few, it was the other way, but not much. But the killer app, I think, is things the customer won't even realize they are getting the benefit of. Because once they have the smart meter, home area networking for things can be enabled. They can have an application on their iPhone that may actually be able to help them do something at home when they are off to work or off on a vacation that they could save significantly on their money or program the charging of an electric car or whatever this is going to be to bring value. But they don't realize the smart meter is in middle of the play there. The Smart Grid, I would say the PMUs, are part of the Smart Grid if you have the head in application that gives you the value. There are some networks looking at, some areas of the country looking at bringing the pleasure measurement units together to do advanced grid control functions. It may be another 10, 20 years before it really works for us, but...
Member: I think there are two trends that are coming together. And the first one probably will play out over the next 50 or 60 years, and that is turning the paradigm that we've run the industry on upside down. So we've always been a load-serving industry. You flip a switch, and we will make more. Probably 60 to 70 years from now, we will be a supplyfollowing industry. We make more, and you take it. We make less, and you do less as we move to more and more variable renewables and other more environmentally friendly sources of energy. So that is going to be critical. The other thing we are finding is the traditional generation transmission distribution customer doesn't work anymore. So the IEEE is in the process of moving to three domains, make, move, and use, because those terms are reasonably easy for the consumers to understand, and it breaks down the old barriers of this is transmission. This is distribution. I make things in generation. I consume them in the consumer area, because that isn't going to be true anymore. We are going to have a lot of consumers who are actually going to make more at certain times than they consume. And I may move electricity from my house to my neighbor's house. I may not actually move it any further than that. And so, you know, we are going to see a lot of changes in the way this industry actually works over the next 50 or 60 years, and I'm glad I'm retiring in another decade.
Moderator: Well, let me put a question to you. In order to make this new paradigm real, how are you going to manage the resources that are involved in the different stages of this new paradigm? Is it going to be like the traditional utility, all centralized, or is there going to be some form of some distributed management so the intelligence of the utilities actually spread out and doesn't come into one central point which tries to control everything?
Member: Well, I think if you’re Oracle or IBM, you are hoping for that centralized control in the ability to put super computers and massive databases into every utility in the world. I think if you are rational, you are going to realize that at least some of that control needs to be distributed now. Do we do fuzzy logic with parameters out in the field and we download different sets of parameters at different times of the day or different days of the week or different seasons so that people operate within local parameters that maybe come from a centralized location? Like people talk about prices to devices. Prices are a set of parameters. I don't know yet. But it scares me to think about how big the data center is going to have to be if we have, you know, a million sets of home appliances all reporting in, and the average home today has 63 or 64 electric consuming devices in it. So that would be 64 million devices, plus a transformer for every three homes, plus sectionalizers, plus automated feeder switches, plus distribution automation, plus substation automation, plus PMU use, plus, plus, plus. Then you have all the people out in the field trying to maintain it. Thank you, no thank you. I don't think I want to centralize all of that.
Moderator: The bandwidth just isn't there to support it, is it?
Member: I think that is the decision point, as a utility, you have to make. Do you want to be behind the customer's wall, or do you want to control the one data point which is the meter? You know, I see further to Kathy's point. I was in Florida last week, and I heard a gentleman from Portland General Electric saying if the grid is so smart, why is my distribution so dumb? Because it is a followthemoney strategy. You have vendors out there pushing home area networks. You have companies like Google and different, you know, Microsoft, they want to get in that market, big, huge aggregators that want to get into that home area network energy management market. And you have meter vendors out there and meter manufacturers, and while that is labor intensive, it is not as labor intensive as going out and making your distribution network smart. That is a labor intensive project. You have to go out, and, you know, you have to change out switches. You have to do a lot of work in the distribution grid. And if you just go into a home and change out a meter, and of course you have to build the TGBs, the base stations, and all of that, but those are centralized types of construction projects. If you are going out to do a distribution upgrade for Smart Grid, you are going to have a lot of guys in there, a lot of trucks rolling out there, a lot of disruptions in your grid. So right now, it seems like the focus, it is a vendor driven followthemoney strategy. And so you build this front end piece at the utility. You build this back end piece at the customer's home. And now you make a decision how far do you want to go into the customer's home here? And now you have these apps developed. If you don't go into the customer's home, does that give you more resources to do the distribution buildout and make the grid truly smart? Those are decision points I see looming in the future. I think plugin electric vehicles is going to be a big game-changer in the west, and all of those things will have to drive grid reliability, distribution modifications, and that type of thing. I don't see that in the near future, though. I see the focus being on these end pieces to gather information.
Moderator: A lot of these devices that are both customer devices and also the devices used to supply power to them have got some intelligence in them right now. They are IP-enabled. In principle, they can talk to each other. In principle, they can be grouped together, and the group can manage itself. And then, that model you got already displays some distributed intelligence. Basically, you are shoving everything towards a Smart Grid without the Smart Grid actually being there. A lot of the components are starting to coalesce by themselves.
Member: So is it going to flow from the substation or back from the customer?
Moderator: That is my question.
Member: That is the question, right, all of the information that the home area networks allow us to gather. We have a relatively small pilot with a huge, you know, projected buildout. But all of the information that we are gathering through the interval data and customer usage enables us, on a broad deployment, to determine the actual load in that pocket of town. So once we get that information, is that the triggering mechanism that brings us into the grid and says, hey, we have got some load issues in this spot. We have got some old feeders. You know, do we put some switches in there that we can control remotely or has sensors that, you know, detect issues and enable or transfer power to another area or shut down equipment in a targeted area? Right now the demand response, we have it is a paging network. Oneway paging, twoway thermostats. We have the ability to call on different device populations, but we can't target a specific area of town that may be an issue. But that is when the grid truly becomes smart, when you can start targeting those areas.
Moderator: Anyone else on this?
Member: RSC has a twoway automatic control response. We do hot water control, and we are moving into air conditioning control to where if the load begins to build in a particular area, we can start cutting off. We maintain 155,000 smart meters. We talk to each one. We are moving as soon as we get the bandwidth, which is part of our DOE project to be able to get those 15minute demand responses from these meters, so we will be able to track the load and compensate appropriately. The other thing is: we use it for restoration. Part of the initiative is, our concern and our focus is, on the member. We want to make sure that they are not without power for any length of time. So we have response units that tell us, along with the SCADA, that we are not getting load here or we have units that call in. That way, we can response with the outage management system all tied together. We know where the problem is and, with the [service] trucks in the field. We are automatically rolling somebody [to do servicing] before the [members] even know. So we are getting information from the field. That is part of our focus being Smart Grid. It is the member focus.
Moderator: So it is interesting listening to this, because it sounds as though utilities - one of the new goals of a utility - is to enable parts of itself to communicate and to provide the bandwidth necessary. And, therefore, to develop a telecommunications plan out of some mix of technologies to supply this bandwidth allowing all of the different parts that the utility manages to communicate with each other and, therefore, to anticipate problems before they occur or to provide information that allows the utility to balance the load that is it experiencing at the time.
Member: Yeah. It is, you can kind of think of it like building blocks, like little Lego bricks. So for some utilities that don't yet have the ability to have a lot of intelligence from the various distribution points, putting in an automatic metering infrastructure is a big component, a big Lego brick, to have this capability. The others, they are focusing more on distribution projects so they have a lot more resiliency in the distribution process. One of the things we are involved with is the DOE Smart Grid investment grant project, SGIG. We are the cyber reps for DOE. So if you are getting an audit on your SGIG project, we are one of the guys that come out and audit the process. And so what happened was, DOE allowed about $4 billion of stimulus money to be released all at once a couple years ago, and utilities had to write up a grant that said, this is how we are going to use the money from DOE to increase our ability to provide a reliable infrastructure to our customer base. So different utilities projected different reasons why they wanted to use the money. Some did AMI projects. Some did distribute wireless projects. A lot of different types of projects were funded through DOE. Then what happened was this big release of money into the infrastructure created a demand for products, such as smart meters, different RTUs, different IP-enabled devices that all of a sudden that really didn't have a chance to go through a proper cyber security checks. So the worst nightmare that Department of Energy had was: okay, now we are funding installations that could actually be the demise of the grid, because we are operating too fast to deploy technology that we don't quite understand how to secure yet. So that is kind of where we are today is that we are forging new ground in the U.S., trying to build this Smart Grid that we don't know what it is going to be yet. We want to make sure that each little piece interoperates with each other in a way that is secure and not that is not going to cause further reliability issues. Because the old model, it was quite you know, you knew where your perimeters were.
Moderator: They were centralized, which is easier.
Member: Yeah. You had: power generation is ‘this’, there are firewalls, I know how to secure this plant. Transmission at a high voltage level. Here's the firewall. This is contained. And distribution was a totally separate system. So now when you are talking about a Smart Grid and a home meter AMI infrastructure that can communicate with energy control centers which communicates with weather systems, which also communicates with marketing building, which Internet portals, which also communicates to low generation systems. Now you are breaking that hardened perimeters and creating this ubiquitous system that if you are not careful could create additional vulnerabilities in the process.
Member: A lot of smart meters, like the ones we deployed, have the remote disconnect. And so you really want to make sure you have a secure connection so that somebody doesn't maliciously start shutting off, you know, all these people, all your customers power.
Moderator: We will be getting into the security aspects. But, you know, the interesting point here is that this seems to be another juncture at which you want to say, well, what is our strategy for managing this whole thing? I mean, all of the IP connectivity allows for the possibility of things managing themselves more towards the edge of the network and being more dynamic and talk. You just plug stuff in. It says, “hi, I'm such and such a device with this IP address, and I'm part of this group here, and we will manage ourselves”. But on the other hand, as a utility manager, you've lost a bit of control. But if you insist on centralizing the control, you run out of bandwidth. And if you add security, that puts more load on the bandwidth. So where does the utility go? What is the feeling? Are we still going to want centralized management? If so, what or are we going to go only distributed.
Member: You need to be a little careful, because it is not an either/or.
Member: And it is not an “everything has to talk to everything”, and you have to break perimeters. It is a careful set of deliberate steps about what you are going to do and what you are not going to do. So, for instance, there is a big project on home area networks where, yes, it is a very decentralized sort of a thing. But nothing is allowed to join if the IP address and the MAC address aren't on a verified list from the manufacturer. And so you are just not allowed in if a central location doesn't already have from the manufacturer the appropriate MAC address information, and so on and so forth, and that has to be verified. Once that is verified, you are allowed to be a part of the network, but the firewall and meter says, nothing from inside of that home area network goes further up the utility communications network. So I will broadcast into the home, and I will take actions to say, “yes, you heard it”, but in terms of other information coming back, “no, sir, none of that information comes back”. And by carefully thinking through the layers and the way things work, you can have a combination of both centralized and decentralized, which is a lot safer to operate and gives you enough control to feel comfortable, but doesn't force you to have, you know, gigawatts of bandwidth, (pardon, wrong term, but you get the idea).
Member: Are you suggesting using another source to receive the data from the customer, from the appliances other than the meter?
Member: Different communications.
Member: You've already have Comcast out there offering to manage and run home area networks through XFINITY.
Member: Comcast, Cox, that whole industry.
Member: That whole industry is very interested in it. AT&T is now beginning to offer along those lines to bring it in through their WOW system. So the utility doesn't have to do anything other than provide pricing information. Now, what these guys don't understand is: I can have 20 homes on a street. They all have different tariffs because they were connected at different times. They have different sized meters, and things of that sort. And so it is until the regulatory side cleans itself up around tariffs, it may be very difficult for Comcast or XFINITY or AT&T or anybody else to send broadcast signals out and say, this is what is happening.
Member: If you get the situation, or you know, certain communities offer single franchise. I was from the Twin Cities, and you could only get if you lived in this community, you could only buy cable from Comcast. But where I'm at now, there are multiple vendors that you could anyway, the point is, how do you start managing this here? Your customers are changing their cable providers at will, you know. If you are going to try to maintain a connection to a customer through another source, like a cable, it would just get messy, I would think, trying to manage that.
Moderator: One of the interesting things I'm hearing here - if I'm hearing correctly - is that where utilities used to basically own or manage almost all of their resources, what I hear now is talking about outsourcing major parts of the communication network and its connections. Is that correct? Is that what I'm hearing?
Member: There is outsource appropriate deals that can be done, right. There are some communications that can be outsourced if aren't critical infrastructure communications. There are others that may require private network. I've got a question for Jonathan, though. The money that was released. You talk about projects being done prior to being, you know, completely cyber security enabled. Do you think that all of that money that was dumped on this market has potentially created a condition where some of the technology was deployed prematurely - before it was quite ready to be deployed…?
Member: Absolutely, absolutely.
Member: …in a rush to get it there, to get it in place?
Member: And all of a sudden you are sitting on this, say, meter network that is out there with this radio coverage that can't provide sufficient bandwidth for all of the applications that are required for energy management.
Member: And there is a theme out there that kind of like the cable industry. I was in the cable industry for years. Every five years we'd rebuild the cable system. We'd upgrade it. We'd go out and cut out all the electronics, put in new electronics. A couple years later we run fiber backbones all of our existing coax. There is a theme out there. Everything we built in the last five years is going to have to be replaced in the next five. And you have got to get out there and touch the customer.
Moderator: You make a distinction between the communications network that the utility considers critical, and then there is the other communications capabilities that you outsource.
Member: I have seen in just in Brazil, the communication. Wherever there is sufficient coverage of, let's say, cellular system or cable system, utilities would try to use them for some of their communication needs. For instance, records list, when you have to close, again, the switch, they can use a device that is powered by our cellular system. They use some cable system, and some of the utilities have their own cable. They are service providers. They are cable service providers. For instance, one in (inaudible) they do it. It is a part of the Smart Grid, but they can't really trust the service provider, for instance, because it will go out much quicker than their own system. And I have seen those things happen consistently.
Member: If you have a huge customer base of 200,000 cable TV viewers and you have a utility which maybe has 50 end points, you are going to focus possibility, prioritize those.
Member: Of course. Same for cellular. Because they have one meter subscriber, and you have only so many technicians on the job - they provide the service to the one million subscribers and not to the places where utility wants to have the coverage.
Moderator: So is this where radio steps in?
Member: Of course.
Moderator: It provides for the critical communications, a utilityowned communications technology, which isn't going to suffer from outages at all. Sorry, I will get Kazimierz and then Dennis.
Member: I think the way to look at this, perhaps, in a better light is through use cases. Define the application by the communications parameters that are needed for the application. Coverage is one thing. Latency is another one. Reliability is another one. And oftentimes, the bandwidth needed to carry the signals is ancillary. I mean, we have critical communications in grid monitoring that consume a very small amount of bytes, but they are very critical. In other areas, you can use a commercial provider for a use case that doesn't require such low latency. For example, Consumers Energy just went with cellular network for their entire meter reading system. Essentially, went with a vendor called Smart (unintelligible.) They felt it was a better solution for them rather than pay the upfront capex cost of building a huge network in an area where they feel cellular has a very robust coverage. They elected to increase opex cost for their metering network and just pay every year instead of taking a huge hit.
Member: Regulatory pressures do that.
Member: For fixed equipment. A meter is a piece of fixed equipment. Yes, it can be done. For mobile equipment, it is not that easy, okay. But another thing, in Brazil, it has happened. I don't see the distribution transmission and generation are really the same. In Brazil, it will not happen in very near future, because the government gave the concessions to separate companies. So, a company that generates really has no transmission. A company that transmits has no distribution. So the distribution company has no generation or transmission usually. Then they have to solve their own problems. And they are different, completely different. And those problems and then it means a complete Smart Grid, something that goes from the house meter down to the generation won't happen like that in Brazil, very foreseeable future. Each of those companies has to solve their own, let's say, electronic problems how to best use the smart features inside their systems.
Moderator: Real complication. Dennis.
Member: You touched on reliability a little while ago. Right now, we were operating on two axis in the meter reading system on Delco, regular full wire circuits. And it got to the point when we had one go out, it may take two weeks to get it repaired. So, in light of what was going on, we switched all of the four wires, upgraded all of the substation units to Raven cellular modems. Now we are reading all of that through Raven. Eventually, the DOE money is going to replace those with their own network. We are big on being able to control the network, the infrastructure. If it is a problem, we know it is our problem, and we get to fix it. We don't have to wait on anybody else to fix it. And that has worked very well. Our outages, we pride ourselves on being able to get a communication outage repaired in two hours or less from the time we know about it. So that is what our goal is, is the reliability. And part of that is [to use] layers. You know, we have a corporate network, and we have a set of VLANs that the Smart Grid will be off to the side. You won't be able to get into the corporate network, because it will be in this the VLAN network, which will be firewalled as it goes out.
Member: I think most prudent operators have this perception that they want to own and control their own infrastructure where it counts. When they need to be able to touch that reclosure, return power very quickly, they want to have control over the latency, the up time, the
Moderator: Well, they get penalized for outages.
Member: Right. And as cellular companies could potentially outsource themselves, what happens if you're leveraging AT&T or Verizon today? They are a company that can outsource themselves to India, Pakistan, China, someone else. So the worst case is, well, now I'm putting my eggs, my basket into someone else's care, and I don't really know for sure if I'm going to be serviced in an adequate amount of time. And then, exactly like you said with the PUCs, I could be fined for something that is out of my control. So I think most utilities have the mindset, if it is something that is very critical, they want to deploy their own infrastructure for something that is monitoring only other customer end use cases might call for allowing another carrier to pick that up for them.
Member: Let's be a little careful here, because, you know, we've leased dry copper for SCADA from the telecoms for 50plus years.
Member: But that
Member: It doesn't matter. We have outsourced that piece of things for years. Satellite communications, we've outsourced the satellites for years. No utility that I know of actually owns the telecommunications satellite in orbit. And on the other hand, AT&T and Verizon have leased fiber from the utilities for decades. And the most important thing is, know your use cases and make sure you have contracts with really good service level agreements and appropriate penalties for the pieces that you don't own.
Member: But a service level agreement doesn't do you any good when you are trying to restore power. It doesn't do you any good to try to go back and collect money after the fact when you don't have the control.
Member: I completely agree it doesn't, but you don't own every piece of the infrastructure you run today.
Member: No, but we have agreements with those who (unintelligible.) We know that they have generator backup and it is a requirement. Whereas, if you are just getting a service, you don't know what is at those particular substations. You don't know there is generation backup at those locations.
Member: Forgive my poor terminology, Kathleen. When I said ‘service level agreements‘ I meant it in the broader level: there will be generators, there will be batteries. Batteries will be tested. Wires will be maintained, and all of the other things, not just latency will be a number of milliseconds.
Member: Things you know, in the commercial industry, my devices seem to work all the time. They are pretty reliable. We have SLA in place, and it is not going to stop the outage. It is going to happen in the utility industry. It is going to happen in the commercial industry. Somebody may skip a maintenance period. Somebody may forget to gas up the diesel generator. Things happen. So I've always, you know, been a proponent of shared services, and the conversation sometimes gets to the point where the utility industry and a lot of been a lot of documentation in this new pole attachment act that has gone out in the last year. The utility industry is putting themselves on this pedestal saying, hey, we are the only ones that can do this. Realistically, if you look at it, the commercial telecommunications industry is where it is today, because they still provide reliable service to their customers. But the point I'm trying to make, and I don't want to expound on it too much, is that things happen. And whether it is you have an SLA in place or whether it is your own people in the field, those outages happen. A tree branch, it is whatever it is.
Member: You need to make sure you minimize those outages, and you can't if a network is down for days. You have to be able to communicate to get your power restored, or nothing is going to come up, because you aren't going to get power to the cell companies to bring up the communications you need to be able to use it.
Member: But they have an interest themselves.
Member: But I think one of the things we haven't talked about is the fact that we, as utilities, we don't have spectrum, you know. And so in an ideal world, where we had spectrum that we would could go out and use ourselves, I think that our networks would look differently than we [have currently] designed them. We designed them the way we do because we don't necessarily have the options that we want to have because we don't have spectrum. We can't afford to go out and buy spectrum at auction. We don't get given spectrum like public safety does. We have to piecemeal your networks together and make partnerships and try to formulate what we can with what we have and make the best situation that we that we can with the resources that we have.
Member: They help, but if the satellite system is down, then the SLA doesn't mean anything.
Moderator: Back to this, too. I mean, there is a part of the utilities which is absolutely critical that they want as much control over as possible. There is an irreducible of nugget of communication that they won't own, because if you outsource everything, if the service provider can't provide the service, utilities may collapse financially. So even if you have SLA, you've got zero redress, and the utility, being a critical infrastructure, can't afford to be left with nothing.
Member: Well, even if their networks are up, you sometimes can't get through, because they are busied out because the general public has over-subscribed to the network, and everybody is trying to phone home during the Hurricane, earthquake, or whatever it is event.
Member: Or when high school gets out.
Member: Or whatever it is happens. So, again, it is use cases. You have got to determine as a utility what falls into what you would classify as mission critical that your CEO is going to have to potentially testify to his regulators or his financial oversight bodies and be able to say, I had control of this situation, or no, I put it in the hands of the carrier because of an SLA, and it failed. How does that feel to that person?
Member: If it is mission critical, you really have to have multiple units of providing that connectivity. Same thing with we face this all the time interoperability. Public safety goes out, and they are going to be build an IP interrupt system, and they are going to rely on what? Charter, Comcast, Verizon? Verizon is predicting, without change, that they will be totally tapped out by February of next year with their capacity. That is on a regular day, never mind when something bad happens and everybody wants to check on the family, you know, and the system just gets swamped and overloaded and crashes.
Member: I just finished a review of transmission SCADA where two different carriers were selected for dry copper from every location, two different routes from every location, and over time, because they bought fiber from the same carriers, and so on and so forth, we ended up with those two different diverse routes actually being on the same fiber in different places. And it takes diligence to make sure that those sorts of things don't happen. It used to be that you could actually walk the lines because they were all overhead, and you could follow them and make sure that they didn't converge. And now they go into a multiplex box and down into the ground, and you have no clue where those things are going. The industry has plowed a tremendous amount of fiber. I know Oncor has put a significant amount of fiber into substations, and so on and so forth, but I also happen to collect a very set of strange photographs. I like to have them. I have literally thousands of photographs of equipment that was used for target practice by hunters. And it doesn't matter what you think everything is going to do on a particular day. If a hunter has put 30 or 40 rounds from a deer rifle through the fuel tank and through the generator in a dog house at a remote location, I'm sorry, it is not going to start when it needs to. And it doesn't matter if you schedule weekly maintenance if they come in on Wednesday, and on Thursday, the hunter does something and on Saturday you have a storm.
Member: Things happen. You've got to have really, really strong contracts. You have to constantly test things. You've got to understand what your use cases are, and you've got to spend money on the critical things. You can't, at this point, afford to spend money on everything that everybody wants you to. The money just doesn't exist. They won't let you raise rates enough. I'm absolutely in agreement with Kathleen. We don't have enough bandwidth in spectrum available to the industry. I do appreciate the fact congress finally said, hey, you can have some of D block over the FCC's objections. And Klaus [Bender], I want to thank the UTC and all the work you guys did to at least get us a toe hold in that space.
Member: It is much appreciated.
Moderator: Spectrum is absolutely critical for utilities right now. I mean, this is something that public safety has known about for some time, you know. They're mission critical. They use radio because they can rely on it. It doesn't matter if there are other options there. This is the core of their service for utilities. If I hear you correctly, Doug, maintenance of what you've got right now is absolutely critical in the communication to make sure that maintenance is done as expeditiously as possible is essential. So this is where, in spite of all of the IP options, radio sits. The question is, is DMR as a digital radio that can work in an IP environment across IP networks. Is this a good thing for utilities or not? And, what can be done to make the best use of the scarce spectrum that is out there? The FCC certainly would have liked, I'm sure, to have managed that spectrum for utilities a lot better, but, you know, it is what it is.
Member: I don't know how it works here, but in Brazil, you have to move into digital for two reasons. Because Brazil FCC requires them to narrowband from from 25kHz down to 12.5kHz. And second, for some reason, it is only known to the Anatel – the Brazilian FCC - they declared that we cannot type approve analog radios from a certain date. Then every single utility in Brazil has to move into digital. And how many digital options there are, there are we already there is P25, there is DMR - two flavors of DMR - and there is Tetra. Tetra is something you don't use in – the United States, but it is available in Brazil. Then, each utility will have to synch it. They cannot do anything else. There is another problem that I don't know if you have here. Yesterday we went into the desert and our cells were still working kilometers from [any population]. But in Brazil, that is not true. You go into into remote areas, you have no cellular coverage, not at all. Then, that is the utilities have to put their own networks. We are just now finishing a commissioning of our transmission utility with 80 towers over some it is thousands and thousands of kilometers of digital network and transmission lines in places where there is no coverage at all for the utility. They need their own communications network, because they have to they are fined by government if something goes wrong, and it doesn't matter if it was a hurricane or if it was vandalism. It happens. They have to serve their market. And that market is much more important. So they need to have communication.
Member: Well, we might have had communications where we were at last night, but there are a lot of places in the United States that we don't have cell phone coverage.
Member: In populous areas, too.
Member: We weren't that far out of town. There are a lot more remote places than we were yesterday.
Member: Small number.
Member: I can't get cell coverage walking down the hallway at the hotel, either.
Member: It is a small number, but anyhow, I'm absolutely sure that you certainly have places, because where the the cellular people go to places where there are people, but your transmission lines will go through, through…
Moderator: …Places where there are no people.
Member: So I have a couple of comments about IP radio systems. So we operate an analog SmartZone Motorola system right now, and we are upgrading to a P25 core, which will be IP-based. And I have, you know, a couple of good things about it being IP-based. One, with a backup control center, we will be able to route automatically if we lose our main control center, we will be able to have our radio system. We will be able to recover your radio system at the backup control center basically automatically. So there is a benefit. However, we had an incident at my company last Saturday when our network, our corporate network, our SCADA network all crashed. They are supposedly not connected. I don't know a whole lot about the network side of it, because I'm not a network engineer. I'm a telecom engineer. I have a lot of questions from my operations people, because our mobile radio system is the last resort for communications. Everything goes down, SCADA communication goes down, but mobile radio has to work, has to be there. We haven't found the root cause of what happened, but they are concerned that if this were to take down our EMS system, took down our corporate network, what would happen to our mobile radio system, also IP, which is supposedly separate? Are they [the mobile radio network and the other networks] still somewhat connected? You know, and so there is a lot of concern about that, and I think that that is going to be I think that is going to be a case for a lot of people when they get into having IP-based radio systems.
Member: We have implemented cellular IP systems, for utilities, for public safety, and for trains. And you have no mission critical no more mission critical than train. Every single communication is going over the radio. And in IP, IP simply is the same as wire, because it is only internal protocol. It is not really connected to the external –networks.
Member: But they are somewhat
Member: They are somewhat. I think it depends how your network is designed. And like I said, our EMS system is supposedly not connected to our corporate network either, and our EMS system also went down. Where those connection points, you know, and how are they connected? I don't know, because I'm not a network engineer. But at some point, they are connected, and when you get instability in your IP network, what is I mean, the question really is by our operation center, what is that going to do to our radio network? And I think right now we I can't answer that. I'm the radio engineer. I'm the one putting it in. I can't answer that. I have no idea what it will do and where those connection points are, because I rely on my network engineer for the design of it.
Member: Listen, but I have no single incident that really something happened because it is an IP system.
Member: But possibly you have a network where they are where they are completely separate and there is absolutely no tie. I think there is a way to do that, as well, so that it truly is its own network that has nothing tied to it.
Member: I think one of the questions that I would have, because we don't do mobile data on our system yet, [is about DMR and AVL]. We are contemplating doing AVL, and I know when we talk about doing work order dispatch for a distribution cooperative, and stuff like that, and the middleware, which is pretty much, I believe, the same thing you are talking about, strips off the sensitive data that changes. I think there is an unknown as to what that cost is. And do I have to have different middleware for different applications. I have 28 distribution cooperatives that I would potentially be supporting, 28 different work order dispatch systems. Do each one of them have to have separate types of middleware? And so [the problem is] trying to figure out how that works, and we are at the very beginning of even looking at the capabilities of putting data onto a radio system. But I think the costs are unknown. And how does that look? And I also hear about you can transfer things from WiFi when you are in your office, and the middleware will route it to your private network when you get into your private network. But, you know, until you start doing a design, honestly I have no idea. Is it $3 million? It is $30,000? I really have not even a ballpark as to what it would be. So that is kind of where I'm coming from as a challenge right now.
Member: I think you will find that it will vary heavily depending on what you are starting from. You know, folks who are starting from older applications that are used to having fat clients where a large percentage of the data lives on the actual client, the middleware is pretty simple and pretty cheap and those sorts of things. And those people who have gone to newer thin clients where you are using a web app, you find out that all of a sudden there is a lot more that you have to deal with and you've got to generate a fat client out on the laptop or whatever device is out in the field. And that becomes a much more expensive way of doing things.
Member: The four conversations I just heard lead into what I was just going to comment on actually. We started with number one, future of the digital connected utility. Then the conversation turned into the present needs of the utility as the utility is today. We talked a lot about communicating with the mobile work force. Communicating with the mobile work forces the way today's mobile work force operates. But tomorrow's mobile workforce is going to operate differently. You are going to have video imaging onsite. If someone does go down, it is not going to be just that warning button. It is going to be, you know, instant communications to the hospital network who have their own safety network in place, visual imaging to the doctor who can make an offsite call as to how to immediately treat that person. So in the utility of the future, the workforce of the future is going to be much more connected and those requirements are going to be bandwidth-intensive. There will be some less bandwidth intensive needs out there, and DMR must have a place for those types of needs. I think about my home area network where we are using the customer broadband to backhaul and download our firmware. It is a 40- 60Mb download of firmware to upgrade that device. We can't do that over our radio network. We won't be able to do that over, I don't believe, a DMRtype network. It has to be done to those devices almost real-time. I mean, you can't take all night to download one device, so it is something to be cognizant of.
Moderator: How big did you say that was?
Member: 40 to 60Mb package for a firmware download is what the vendors told us when we do the firmware.
Moderator: They should rewrite their firmware.
Member: Big applications. These are the applications. I'm not sure what they are using them for, but the deal we have with the device manufacturers and eventually they will be offering energy management applications to the customer. The point I'm making is: today's utility doesn't look anything like tomorrow's utility will, and tomorrow's utility, in my observation, is going to require a lot of bandwidth.
Member: I think you also have to be able to get that bandwidth in order to provide those applications. And if we don't have access to that bandwidth, you may not be able to have those applications.
Member: You won't have access to it. There are other people that will have that bandwidth right now, so it is the hybrid network that needs to be.
Member: If that is the key, it is sort of the classic use case. I can see a mobile worker who has a use case for which DMR is the perfect fit. That is mission critical voice. If you look at the fact that there are a number of things happening coincidentally inside DMR. I don't know that DMR is necessarily driving the bus, but it certainly is going along for the ride. You’ve got enhanced intelligence of the network operation, which means, as I alluded to earlier, that you can better utilize what spectrum you have and point things in the right direction at the right time so you can make intelligent decisions with what your payloads are that didn't exist before. And to mention a competitor, you may know that recently Kenwood and JBC merged, and JBC brought to the table some very interesting video solutions that I've seen run perfectly from Japan to the U.S. over 6.25kHz digital radio. Now, it is not real-time. It is not, you know, streaming video, but the resolution and stuff is incredible. The quality of the pictures is incredible. And for things like security where you want to visually see, for example, maybe they got the codes, or whatever, but is it really the employee that you think it is that is trying to enter that substation? There is a whole host of new tools that this technology and the other aspects of it that are coming along, concurrently, that will be available for you to shape and address your needs going forward. Maybe they exist already, but you really knew there was nothing there to deal with them, so you would say “we will put those aside”. But certainly, a lot of those are being brought to a head, particularly with enhanced security and stuff. And I think it is kind of an exciting time. You are going to have tools that you never had before. So you need to start thinking somewhat outside the box as to how you can, you know, merge all of this and together into a cohesive solution to your needs. Because, as Kathy pointed out, obviously, every entity’s needs differ somewhat from everybody else's.