PB was commissioned by Saudi Electricity Company—Eastern Region Branch (SEC-ERB) in February 2001 to carry out a technical review of its existing SCADA communication network and to identify new communication concepts that are more flexible in terms of expansion and reconfiguration, and that have higher bandwidth potential whilst retaining the qualities essential for SCADA operations. SEC-ERB has about 7000 employees and generates 8000 MW to 0.5 million customers.

Our review was specific for an electricity utility, but the principles proposed for migration to transmission control protocol/internet protocol (TCP/IP) and Ethernet networking are equally applicable to other industries employing SCADA systems.

What is Driving the Need for Change?

A traditional SCADA network guarantees that a communication path that is always there and that transmission delay and variations are very small, but the technology is becoming obsolete and unsuited to today’s power system operations. In recent years, telecommunication systems have undergone radical change prompted mainly by the desire to increase system performance.

At the same time, deregulation and privatisation of the electricity industry coupled with the liberalisation of the telecommunication market have imposed new requirements on electrical power communication networks. Drivers for change include plans to reduce workforces and take advantage of new revenue opportunities, along with the following key technical drivers.

Optical Fibre Installations. Optical fibre technology is the most powerful and versatile communication medium due to its high bandwidth capability and immunity from electromagnetic interference. The use of optical fibre has become commonplace in most electricity utilities.

TCP/IP Networking. The protocol suite and associated technologies that are used to deliver the Internet and corporate intranets use connectionless networking. IP datagrams are transmitted across a network on a best efforts basis. TCP/IP has not been ideally suited to the transport of SCADA traffic because transmission times may vary and performance under extreme conditions may not be predictable.

Emergence of QoS over TCP/IP. A great deal of development work has taken place to produce Quality of Service (QoS) standards that will guarantee delivery of connection-oriented traffic over connectionless TCP/IP networks. The Internet Engineering Task Force has studied options for providing QoS on TCP/IP networks and two schemes are emerging—Diffserve and Intserve. The two schemes are not mutually exclusive.

• Diffserve (Differentiated Services) is where the network controls the quality of service by allocating priorities to data types. Differentiated services uses Multi Protocol Label Switching (MPLS) to deliver QoS. It proposes three classes of data transfer—expedited, assured and best efforts forwarding. MPLS is supported by a number of major network equipment suppliers. Data frames are marked as near the network edge as possible with a suitable label that indicates priority. The core network Label Switch Routers can then treat the frames appropriately. MPLS can be used for setting up virtual private networks (VPNs).
• Intserve (Integrated Services) allows the end system or application to request a QoS from the network using Resource Reservation Protocol (RSVP). The network must then set up facilities to provide the requested QoS. Intserve is an end-system-based service. It proposes three levels of service—guaranteed, controlled and best efforts. In practice, RSVP has been seen to be slow in setting up connections and it may not scale well. RSVP also gives control to end systems and, hence, can lead to many claimants for the highest priority.

Whether using Diffserve or Intserve, QoS cannot be guaranteed on Ethernet LANs. With 10 Mb or 100 Mb/s available on each segment and a SCADA usage of 9600 b/s or less, this will not be a problem.

Flexibility and Simplification in Communication Network. Conventional SCADA network designs rely on the predictable nature of connection-oriented services using fixed audio bandwidth links, analogue modems and specific protocols. Setting up and maintaining these networks calls for specialised skills. Bandwidth is limited to 3 KHz, which is adequate for current remote telemetry units (RTUs) but potentially limiting as businesses move towards increasing use of substation automation and remote management. The support of analogue modems will become increasingly difficult as the world moves to digital communications.

Changing to digital networking will enable the management of SCADA networks to be integrated into a system common to the main corporate network. Reconfigurations will be simplified to keyboard commands rather than rewiring at multiple points. Bandwidth can be allocated as required and RTUs themselves remotely managed.

Why Move SCADA Networking to TCP/IP?

TCP/IP has become the de facto international standard. The advantages of TCP/IP networking are:

• Worldwide adoption (e.g. the Internet)
• Very well developed hardware and software market
• Simplicity and choice of application layer protocols
• Inherent resilience of the IP routing concept
• Strong network management, including remote control and monitoring.

Using TCP/IP and the commonly associated Ethernet technology will give power system operators access to a wide range of standards-based inexpensive hardware and a large pool of trained staff.

TCP/IP has not been widely introduced into wide area SCADA communication systems to date. This is partly due to the replacement cycle but also because of the non-deterministic nature of Ethernet and TCP/IP communications. The development and implementation of QoS standards removes the risks associated with the underlying connectionless networking. As TCP/IP and Ethernet support is becoming available from SCADA equipment manufacturers, the use of
wide area TCP/IP networking from Master Stations to RTUs is practicable.

SCADA applications use a very wide selection of application level protocols. Two standards are emerging that are designed to operate successfully with TCP/IP and Ethernet:

• DNP3 (Distributed Network Protocol 3), which is gaining acceptance in the North American market
• IEC 60870-5-104, which is favoured in Europe.

Figure 1: TCP/IP Communication Network

Figure 2: Layer Network Topology

Migration to a Single Integrated Network

TCP/IP networking presents the opportunity to migrate to a single network for both operational and non-operational requirements. In addition to business data, other services that can be supported by the TCP/IP networking include the following:

SCADA Data. SCADA equipment can be connected to suitable LAN segments that are linked by the corporate WAN. A speed of 100 Mb/s is recommended as the costs for 10 Mb/s and 100 Mb/s are only marginally different. Having made the Ethernet to Ethernet TCP/IP connection across the network, it will be necessary to engineer adequate performance using QoS.

Video Transmission. The use of closed circuit television (CCTV) for remote security monitoring of substations and power plants is being employed by many electricity utilities. Two options for putting these services on the TCP/IP network are simple Webcams or a full video monitoring service with remote access controls.

Voice Communications. Migration of voice services onto the TCP/IP network can be achieved in stages:

• PABX (private automatic branch exchange) to PABX connection over the IP network.
• IP telephony, which is still developing. Deployment requires a complex set up of call managers, IP telephones and QoS, but it promises to remove the need for separate PABXs.

Substation Automation. Electricity utilities have been considering for some time substation integration with networking and intelligent electronic devices (IEDs) to manage power networks more effectively. Substation
automation is likely to be closely integrated with SCADA and protection systems. Communication network services will be vital to the integration of data acquisition, control and protection.

Electricity utilities may wish to develop TCP/IP networking capability, such as the one shown in Figure 1, to support their power system operation and administrative functions.

Technical Proposal For a TCP/IP and Ethernet Networking

Network Topology. To make the best use of existing networking hardware and available bandwidth, and to meet the high availability requirements of SCADA system, we have suggested that SEC-ERB follows a network topology based on a layered approach with an appropriate level of circuit and equipment diversity. The network should consist of four layers; core, distribution, access and users, as shown in Figure 2.

Core Network Layer. The core network layer is comprised of ATM or gigabit Ethernet switches, routers and the interconnecting data links. The core network will transport data between points on the distribution layer. Full diversity of routes between core layer switches must be provided and data link speeds should be at least E1 (2.048 Mb/s). Core network devices will be MPLS Label Switch Routers.

Distribution Layer. Connection from the core layer to the Distribution layer will be by digital data links at speeds up to E1. Each distribution layer location will be connected to two separate core switches. Distribution layer routers will perform the Label Edge Router function of MPLS. Most RTUs will be connected to the distribution layer.

Access Layer. Access routers will be connected to the distribution layer using suitable network links. These will be 64 Kb/s but where video services are envisaged an E1 circuit may be more appropriate. The access routers will be situated in substation and office locations.

Users. LANs (such as PCs, file and print servers) will be connected to the access layer directly or via local LAN switches. SCADA RTUs will use LANs that are directly connected to distribution layer. SCADA master stations will have diverse connections to two distribution layer devices.

High Availability Design. High availability networking service can be provided to SCADA services (and others). To give the best availability, we have suggested that SEC-ERB follows design guidelines we provided that minimise the effect of a single node or circuit outage.

QoS Recommendation. We have recommended to SEC-ERB that it adopts the internationally standard based Diffserve and MPLS as the method of providing QoS on its digital communications network. SEC-ERB will need to change from the proprietary inter-router protocol EIGRP as MPLS requires the use of Open Shortest Path First (OSPF), a recognised international standard.

Network Security. To manage the threat from external sources most (if not all) operators of private TCP/IP networks use a secure gateway to manage the connection. This gateway is usually called a “firewall” and will often incorporate a “demilitarized zone” (DMZ).

SCADA traffic is effectively segregated from other networked applications when it is carried on analogue circuits. When all traffic is carried on TCP/IP, it will be necessary to provide specific security controls to prevent unauthorised staff from accessing data. The simplest method for achieving segregation is to use the facilities of MPLS to build VPNs. MPLS VPNs are easy to administer and provide any-to-any communication within a community group.

Networking security is designed to work in conjunction with application security and does not replace it. If resale of TCP/IP networking to other organisations is proposed, then the MPLS VPN (Community Group) will be a suitable means for delivering multiple autonomous networks on the same infrastructure.

TCP/IP Addressing. The proposed change to TCP/IP networking for SCADA systems and possibly voice and video may have an impact on an organisation’s TCP/IP addressing plan. Use of the private Class A address (10.0.0.0) will give the greatest address range and flexibility.

The use of MPLS VPNs provides a suitable method for adding “other” users to the network infrastructure and segregating their traffic from each other. “Other” users addressing plans
can be accommodated without change as long as they remain within their own MPLS VPN.

Network Management. A number of network management systems are commercially available. All use Simple Network Management Protocol (SNMP), the most suitable protocol for managing TCP/IP networks. Operators should consider using SNMP to manage RTUs and manufacturers should be requested to supply suitable Management Information Bases (MIBs).

The TCP/IP network will be carrying operationally critical SCADA data, so electricity utilities must ensure that network management support is available 24 hours a day, 7 days a week.

Testing and Staged Implementation of Digital Network. The implementation of QoS on TCP/IP networks is relatively new in the electricity industry. It requires a well planned and systematic sequence of testing, installation and commissioning to migrate operational and business requirements from existing systems.

Electricity utilities are encouraged to test the performance of their detailed design proposals to verify the QoS configuration can deliver the required performance characteristics and that they have full confidence in the concept before it is implemented network wide. To achieve such confidence requires a proving period of off-line network testing and a staged migration implementation.

Potential Business Benefits of Implementing a Single Network

Moving to an all TCP/IP network will enable electricity utilities to select equipment from a very wide range of compatible types. There is potential to use business leverage to obtain best possible prices for this widely used technology.

The sale of network capacity in the form of spare fibres has been available for some time, but this represents the crudest and potentially least profitable option. Resellers will need to consider selling managed data services to third parties in order to maximise profitability. Demand for this type of service is limited to TCP/IP networking. It can best be achieved using MPLS VPNs in the short term and through the development of full traffic engineering services in the future. Entry into this field will depend on market conditions.

Conclusions

There is a significant benefit in migrating SCADA systems to TCP/IP and Ethernet networking. Many power system operators already have the technical infrastructure, capability and capacity to develop a successful migration of SCADA networking to TCP/IP. In addition, TCP/IP networking with QoS has the technical capability to support other power system operational requirements as well as business administration.

Electrical protection systems have critical requirements regarding reliability and performance of communication links, so we do not recommend transfer of teleprotection signalling to TCP/IP networks until such time that connectionless networks have matured and can guarantee the required service for electrical protection systems at all times.