MACsec Intel FPGA System Design

User Guide
Updated for Intel® Quartus® Prime Design Suite: 22.4

Online Version ID: 767516

Send Feedback Version: 2023.03.03
Contents

Contents

1. Introduction................................................................................................................... 4
1.1. Terminology..........................................................................................................7
1.2. Release Information...............................................................................................7
1.3. System Requirements............................................................................................ 7
2. Architecture.................................................................................................................... 8
2.1. System Architecture...............................................................................................8
2.2. Data Path Between Ethernet MAC and MACsec......................................................... 10
2.2.1. Ethernet Subsystem................................................................................. 10
2.3. Data Path Between MACsec and MCDMA................................................................. 14
2.3.1. AXI-ST Multi-Segment to Single-Segment Conversion................................... 15
2.3.2. MCDMA.................................................................................................. 16
2.4. Data Path Between MACsec and Packet Generator/Checker (Packet Client).................. 19
2.4.1. Packet Client........................................................................................... 20
2.4.2. Packet Generation and Check.................................................................... 20
2.5. Data Path Illustrations.......................................................................................... 21
2.5.1. MACsec Interface Signal Names................................................................. 23
2.6. Interrupts........................................................................................................... 24
2.6.1. MCDMA MSI-X Table Configuration..............................................................25
2.7. Packet FIFO........................................................................................................ 25
2.8. AXI-ST Rate Controller......................................................................................... 26
2.9. Error Handling..................................................................................................... 27
2.10. Top Level Signals............................................................................................... 28
3. Interface Overview....................................................................................................... 29
3.1. Clocking............................................................................................................. 29
3.2. Resets................................................................................................................30
4. Parameters................................................................................................................... 32

5. Configuration Registers................................................................................................ 33
5.1. System Register Map............................................................................................33
5.1.1. Packet Client Register Map........................................................................ 34
5.1.2. Interrupt Controller Register Map............................................................... 40
6. Software Architecture................................................................................................... 42
6.1. MACsec Key Agreement Protocol............................................................................ 42
6.2. Driver Functional Requirements............................................................................. 43
6.3. Software Requirements.........................................................................................43
6.4. Software Overview...............................................................................................43
6.4.1. McDMA Driver Kernel Module..................................................................... 45
6.4.2. MACsec IP APIs....................................................................................... 46
6.4.3. Linux MACsec Driver Kernel Module............................................................ 48
6.4.4. IP Tool....................................................................................................48
6.4.5. WPA Supplicant....................................................................................... 49
6.5. MACsec IP APIs Sequence..................................................................................... 49
6.5.1. MACsec Initialization Sequence.................................................................. 49
6.6. Functions............................................................................................................57
6.6.1. macsec_initilize....................................................................................... 57

MACsec Intel FPGA System Design User Guide Send Feedback

2
Contents

6.6.2. macsec_get_attributes............................................................................. 58
6.6.3. macsec_get_sa_attributes.........................................................................59
6.6.4. macsec_set_attributes..............................................................................59
6.6.5. macsec_set_sa_attributes......................................................................... 60
6.6.6. macsec_read_register.............................................................................. 61
6.6.7. macsec_write_register..............................................................................61
6.6.8. macsec_set_port_configuration..................................................................62
6.6.9. macsec_rate_configuration........................................................................62
6.6.10. macsec_single_or_multi_port.................................................................. 63
6.6.11. macsec_crypto_mode............................................................................. 63
6.6.12. macsec_port_priority.............................................................................. 64
6.6.13. macsec_register_isr................................................................................64
6.7. Software Tools.....................................................................................................64
6.7.1. IP Tool....................................................................................................65
6.7.2. WPA_Supplicant.......................................................................................66
6.7.3. CLI Debug Tool........................................................................................ 67
7. Generating the System Design...................................................................................... 72
7.1. Software Requirements.........................................................................................72
7.2. Obtaining the Reference Design............................................................................. 72
7.3. Reference Design Directory Structure..................................................................... 72
7.4. Simulation Command Arguments........................................................................... 73
7.4.1. Simulation -d Options............................................................................... 74
7.5. Simulation Test Cases...........................................................................................75
7.6. Complete Simulation Command............................................................................. 78
7.7. Simulation Requirements...................................................................................... 79
7.8. Running the Simulation ........................................................................................79
7.9. Building, Installing, and Running the Software......................................................... 81
8. Document Revision History for MACsec System Design User Guide............................... 86

Send Feedback MACsec Intel FPGA System Design User Guide

3
767516 | 2023.03.03

Send Feedback

1. Introduction
MACsec was standardized in 2006 by the IEEE (standard IEEE 802.1AE-2006) as a
point-to-point security protocol providing data confidentiality, integrity, and origin
authenticity for traffic over Layer 2 links in a larger security ecosystem for ethernet
LAN. In MACsec, packets flow over unidirectional secure channels (SC), which are
supported by secure associations (SA). The secure associations each use a separate,
randomly generated key.

Figure 1. Ethernet LAN with Number of Channels

0:3
SA 0:3
SA
SC TX
Host 3 SC 1T
SC2
1R X RX
X SC2
c3 Host 2
SA
cse
ma

ma
SA 0:3
cse

0:3
c2

macsec1

SC0 TX SA 0:3 SC0 RX SA 0:3

Host 1 SC0 RX SA 0:3 SC0 TX SA 0:3 Host 4
SC1 RX SA 0:3
SC1 TX SA 0:3
SC2 RX SA 0:3
SC2 TX SA 0:3

As shown in the figure above, each node has at least one unidirectional secure channel
(transmitter to receiver). Each secure channel is associated with an identifier called
Secure Channel Identifier (SCI). Each node, which expects to receive the traffic sent
through a particular transmit secure channel, must be configured with a matching
receive secure channel. This receive secure channel must have an SCI corresponding
to the SCI of the transmit secure channel of the peer. Control logic that maintains the
key look-up tables stores the keys based on SCI. If the incoming packet does not have
the optional SCI field, then the receiver MACsec frame uses a local SCI with the
received destination MAC address along with a fixed port number.

Within each secure channel (both transmit and receive) secure associations are
defined. Each secure association has a corresponding Secure Association Key and is
identified by the Association Number (AN) field of the SecTAG header. Secure
associations have a limited duration, hence both sides need to establish a new secure
association and switch to it once the old one expires. This is called key rotation.
MACsec 802.1AE protocol does not cover the key exchange between a key server
within the LAN and any key client. There is another standard defined for this called
"IEEE 802.1X for port-based network access control".

Intel Corporation. All rights reserved. Intel, the Intel logo, and other Intel marks are trademarks of Intel
Corporation or its subsidiaries. Intel warrants performance of its FPGA and semiconductor products to current
specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any ISO
products and services at any time without notice. Intel assumes no responsibility or liability arising out of the 9001:2015
application or use of any information, product, or service described herein except as expressly agreed to in Registered
writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying
on any published information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
1. Introduction
767516 | 2023.03.03

The figure above also indicates that on a receive node, a host (Host 4 in this example)
can have multiple secure channels as the number of transmit nodes which can
communicate with it (for ease of understanding, consider a dummy switch in
between). This ensures that Host 3 would not be able to decrypt the communication
between Host 1 and Host 4. Similarly Host 1 can’t decrypt the communication between
Host 4 and Host 2 because of their secure channel number differentiation.

Figure 2. An Ethernet Frame Before and After MACsec Processing and Encryption

ETH Payload
GCM-AES
ETH Sec TAG Payload (Encrypted) ICV

TCI/AN SL Packet Number SCI (Optional)

A MACsec packet starts with an Ethernet header with an EtherType of 0x88E5. This is
followed by the MACsec SecTAG, which contains information that helps the receiver
identify the decryption key, as well as a packet number (for replay protection). Within
each secure association, replay protection can be performed by checking the Packet
Number field of the SecTAG header against the packet number locally incremented.
For strict reception ordering and replay protection, the replay protection window is to
set to 0. A non-zero replay window is necessary to support the use of MACsec over
provider networks that reorder frames. Frames within the window can be received out
of order. Each MACsec packet has a unique sequential packet number and each packet
number can only be used once in a given secure association. A secure association
retires once the packet number reaches the maximum possible or programmed value.

After the SecTAG comes the payload, which can be encrypted, and the ICV (Integrity
Check Value), which is compared against a re-generated one by the crypto algorithm
being used to guarantee that the packet is indeed created by a node which is in
possession of the key and has not been modified on the way.

Send Feedback MACsec Intel FPGA System Design User Guide

5
1. Introduction
767516 | 2023.03.03

Figure 3. Top Level Block Diagram of Reference Design

macsec
LAN

VM0 (MACSec0) VM1 (MACSec1)

SC TX SA 0:3 SC TX SA 0:3
SC RX SC RX Host VM 0 VM 1 **Bypass Hypervisor
SA 0:3 SA 0:3
(Key Client) (Key Server)
SW

P-Tile HW
Host Interface
Interrupt
Control MCDMA PIO [vf_active,
clog2 (PF_NUM),
clog2 (VF_NUM),
Channel Decoder (CH0-PF 0, CH1-PF 1) PIO BAR2 Address

AVST AVST
AXI-AVST
FIFOs
CSR *EAPOL
Rate Cnt Pkt Parse

Uncontrolled CSR Crypto ICA

CSR
AXI-AVST

E-Tile TX Ports AES HIP

AXI-ST MACSec0 Packet
RX Rate Cnt Generator
AXI-ST
& Checker
MAC

QSFP
AXI-AVST
Loopback
FIFOs
CSR *EAPOL
Rate Cnt Pkt Parse

Uncontrolled CSR Crypto ICA

CSR
AXI-AVST

E-Tile TX Ports AES HIP

MACSec1 Packet
RX Rate Cnt Generator
AXI-ST
MAC & Checker

The MACsec system level example design provides you with a starting point for your
application development. It can help accelerate your design cycle and enable you to
invest your resources to add unique value to your product. Currently, only the MACsec
IP is released in Quartus without any integrated MACsec software. When using the
stand-alone MACsec IP, you need to integrate it with your own MACsec software for
both the data path and control path to make your solution functional.

This design demonstrates the key integrated components of Intel Agilex 7 FPGA which
supports IPs such as PCIe Endpoint HIP, MCDMA, Crypto HIP, MACsec and Ethernet
MAC HIP. It presents the example of an inline MACsec function between two host
systems on a LAN connected through the QSFP link with Ethernet MACs configured to
support 25G or 100G. The whole system is arranged in a single development kit
(DevKit) which supports multiple MAC and MACsec IPs. A Host machine connected
through the PCIe interface is configured as two Virtual Machines (VMs) tied to two
PCIe PFs (physical functions), each driving the data and control paths independently.
Software running on the host machines also enables CLI (Command Line Interface) for
any user-specific testing.

MACsec Intel FPGA System Design User Guide Send Feedback

6
1. Introduction
767516 | 2023.03.03

1.1. Terminology
Table 1. Terminology
Terminology Description

DMAC Ethernet MAC Destination Address

SMAC Ethernet MAC Source Address

SA Security Association

SC Secure Channel

PN Packet Number

PPBB Packet Processing Building Block

SADB Security Association Database

ICA Inline Crypto Accelerator

Common Port Provides the interface to the Ethernet IP. This port carries traffic from/to both the
Controlled and Uncontrolled ports. The MACsec IP provides up to 64 Common
ports.

Controlled Port A virtual access point which provides full access to the network. This traffic
requires encryption or decryption. The MACsec IP provides up to 64 Controlled
ports.

Uncontrolled Port A virtual access point which provides full access to the network. You must filter
the required packets on this port. In the case of the reference design, EAPOL
packets are filtered. This traffic does not require encryption or decryption. The
MACsec IP provides one Uncontrolled port.

MKA MACsec Key Agreement

EAPoL Extensible Authentication Protocol over LAN

WPA Wi-Fi Protected Access

MCDMA Multichannel Direct Memory Access

1.2. Release Information

The current release of the MACsec Intel FPGA System Design is 1.0.

1.3. System Requirements

Software Requirements
• MCDMA Driver for linux
• MACsec Driver for linux
• SW application that manages MKA and user application flow

For more details, refer to Software Architecture on page 42.

Send Feedback MACsec Intel FPGA System Design User Guide

7
767516 | 2023.03.03

Send Feedback

2. Architecture
This section includes components for the MACsec system design architecture.

2.1. System Architecture

The main design components of this reference design are chosen considering the
reusability which reduces the time to market of your end product.

Figure 4. Detailed Block Diagram of Reference Design

PF1
Port Enable
w.r.t PF/VF

••••
Custom RTL Logic PF0
Quartus IP PF0 •••• PF1
Packet
Inter- CSR Generator Interconnect
connect ICA
& Checker HIP
Cypto
AES
AXI-ST
AVMM
Bar AVMM to

AXI-ST
Clock Rate CTL
Bridge Interpreter AXI-Lite
CSR
Controlled Port CSR
PIO Port MUX/
CDC Packet Multi Seg DeMUX
Common Port

F Drop/Ctrl to Single Port AXI-ST P0

I (Packet Seg &
Uncontrolled Port

VM1 F Filter) Width Conv MACSec0 MUX/ Pkt

(Server)
O
AXI-ST DeMUX FIFO
F Pkt Rate
I
F FIFO CTL
O E/F-Tile QSFP
Loop-
MCDMA

AXI-St AXI-St
P-Tile

to AVST MUX/
Bridge DeMux F back
I Packet Multi Seg CSR
Uncontrolled Port

F Drop/Ctrl to Single
O (Packet Seg &
VM0 Filter) Width Conv AXI-ST
Common Port

(Key F Port AXI-ST P8

I MACSec1
Client) F Pkt Rate MUX/ Pkt
O DeMUX
FIFO CTL Port MUX/ FIFO
DeMUX
Host Controlled Port
Ethernet SS
AXI-ST

SW HW Rate CTL
AXI-ST

Cypto
AES
Packet ICA
Generator HIP
CSR

& Checker

In this reference design, the same QSFP loopback acts as LAN for both MACsec control
and datapath transfers. The MACsec protocol provides two different standards for data
and control transfers respectively: IEEE 802.1AE and IEEE 802.1X. Two Ethernet MACs
are enabled with 2 different MAC addresses connected and controlled through two
host VMs and get data from two different Ethernet MAC ports.

Intel Corporation. All rights reserved. Intel, the Intel logo, and other Intel marks are trademarks of Intel
Corporation or its subsidiaries. Intel warrants performance of its FPGA and semiconductor products to current
specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any ISO
products and services at any time without notice. Intel assumes no responsibility or liability arising out of the 9001:2015
application or use of any information, product, or service described herein except as expressly agreed to in Registered
writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying
on any published information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
2. Architecture
767516 | 2023.03.03

The data path is offloaded to FPGA hardware but controlled through the CSR interface
from host VMs. MCDMA enables the CSR path through the PIO interface. The same
PIO is driving all the components in the two MACsec paths. The FPGA Platform
Designer (Standard) fabric interconnect takes the responsibility of driving to correct
VM path as the PF number is also part of the PIO AVMM address. Before any data
transfer starts, the MACsecs need to be configured with appropriate keys for each
supported security association in a given secure channel along with the SCI info for
look up which contains the destination MAC address. All these are programmed into
the MACsec IP registers using the MACsec driver in the Linux software stack. The
MACsec IP driver acts as an underlying layer for Linux’s default MKA protocol stack
(WPA). Different machines on the LAN interact and elect one machine as Key Server
based on a priority number. The same machine is responsible for providing keys to all
the clients over the same LAN.

In this reference design, one of the 2 VMs acts as a key server and the other acts as a
key client to configure the keys for the MACsecs in the datapath using Linux’s default
MKA protocol stack (WPA). Once the keys are written to the MKA stack, the underlying
MACsec driver writes the keys into the respective registers inside the MACsec IP
through the CSR interface.

Once the keys are established, the application running in a VM can start sending the
Ethernet data packets from the Packet Generators continuously or in one-hot mode
depending on its configuration. The MACsec and Crypto IPs inline in the datapath
provide encryption to the packets. The Ethernet MAC and PHY transmits these
ciphertext packets by appending the required preamble + CRC over the QSFP. On the
other side, these ciphertext packets are received through another Ethernet MAC port
and the data is fed to the MACsec core (RX) attached to the particular MAC to decrypt
the packet. The encryption and decryption parameters are managed by the MACsec
stack running on the host machine.

The MACsec IP in the datapath monitors the 32-bit packet numbers received or
calculated internally and informs the Host VM through an interrupt or poll mechanism
once it reaches the terminal/limit count. The security association (SA) needs to be
changed to the next one and new keys are chosen for the datapath which is called
"key rotation". The MCDMA in the host path enables the communication between
various VM-clients and MACsec’s uncontrolled ports. It operates on queues set up by
the MCDMA driver software to transfer the data between the local FPGA and the host.
The elements of each queue are descriptors. The MCDMA’s control logic reads the
queue descriptors and executes them. Separate queues are used for reading and
writing DMA operations for each channel. Each PF consumes one MCDMA channel in
this design. The logic in the FPGA fabric needs to route the packets based on their
channel ID sideband signal from the AVST streaming interface. The channel adaptor
logic takes care of packet routing in both D2H and H2D directions for 2 PFs.

The host offloads the data transfer overhead to MCMDA for transferring EAPOL packets
or any non-SecTAG packets between the host and MACsec’s uncontrolled ports. In
MCDMA, once configured with channel specific BDs, as soon as it sees a packet in the
packet buffer, it forms a PCIe TLP for the actual length of packet received and sends it
to the host. It may consume multiple BDs if the Ethernet packet length is higher than
the MPS value configured in the PCIe EP. In addition to that, it also updates the BD
status and hands it over to the SW control once served. The MACSec uncontrolled
ports get the plaintext as well as ciphertext data. It is your responsibility to filter the
packets and decide on which packets go to the host.

Send Feedback MACsec Intel FPGA System Design User Guide

9
2. Architecture
767516 | 2023.03.03

The Ethernet MAC instance can be a direct E-Tile/F-Tile Ethernet IP or HSSI-SS IP

based on the FPGA device/Devkit chosen. The choice of HSSI-SS is better (provided
the subsystem is available for the targeted tile) as it provides a standard AXI
streaming interface and is therefore tile-agnostic. In this document, the terms "E-
Tile/F-Tile" and "HSSI-SS" are used interchangeably. The MACsec IP also supports the
AXI streaming interface so it’s easy to connect. The user logic in RX path, between
MAC and MACsec can check the packet status from the specific TUSER bits (for
example, if the CRC error bit is asserted) to decide whether to drop the packet or send
it to the MACsec IP. There is no need to filter the Ethernet packets based on
destination MAC with your logic as the same check happens inside the MACsec IP.

There are multiple protocol bridges and CDC bridges in the design to accommodate
the data transfers between AXI streaming and AVST, and between different clock
domains. The details are discussed in the later sections.

The reference design is flexible enough to use one of the MACsec paths with external
test equipment as the key client/server based on the priority setting at the SW stack
level for interoperability testing. At the packet client interface, you can enable the
loopback and use a 3rd party tester as a pattern generator.

2.2. Data Path Between Ethernet MAC and MACsec

When the Ethernet MAC is configured for 25G/100G, the IP enables the user interface
with 64/512 bit data bus respectively. This system reference design scales to 100G full
duplex support for Devkits once they are available. The packet generator-checker
block supports 100G data rates. Any adaptors in the datapath are modular to achieve
these rates. Future releases may include the HSSI Subsystem instead of the E/F-Tile
QHIP. This path uses the standard signals from both IP interfaces.

2.2.1. Ethernet Subsystem

The E/F-Tile Hard IP is configured in MAC+PCS layer mode. The IP core handles the
frame encapsulation and gives out data from the FPGA at the 25Gbps, or 100Gbps line
rate. For supporting both 25G and 100G, two different IP instances for 25-Gbps and
100-Gbps are used. One of them is selected at the compile time based on the speed
selected. In the transmit direction, the MAC accepts the encrypted frames from the
MACsec and inserts the inter-packet gap (IPG), start control, term control, preamble,
start of frame delimiter (SFD), padding, and FCS bits before passing them to the PHY.

Figure 5. TX Frame Structure

Added by MAC Added by MAC

Start Preamble SFD Destination Source Type/ Payload Pad FCS Term Control IPG
Control [47:0] [7:0] Addr[47:0] Addr[47:0] Length[15:0] [<p>-1:0] [<s>-1:0] [31:0] [7:0] [<g>-1:0]

In the receive direction, the PHY passes frames to the MAC. The MAC accepts frames
from the PHY, performs checks, updates statistics counters, strips out the CRC,
preamble, and SFD, and passes the rest of the frame to the user interface that
connects to the MACsec IP.

MACsec Intel FPGA System Design User Guide Send Feedback

10
2. Architecture
767516 | 2023.03.03

Figure 6. RX Frame Structure

Removed by MAC Removed by MAC

Preamble SFD Destination Source Type/ Payload Pad FCS Term

Start [47:0] [7:0] Addr[47:0] Addr[47:0] Length[15:0] [<p>-1:0] [<s>-1:0] [31:0] [7:0]

2.2.1.1. Data Receiving from MACsec to E/F-Tile Hard IP

The E/F-Tile Hard IP for Ethernet supports the AVST protocol and the MACsec IP
supports AXI-ST. So, the AXI-ST to AVST conversion is done before giving to E/F-tile
Hard IP.

Figure 7. Timing Diagram for Packet Receiving at E/F-Tile Hard IP from MACsec
i_clk_tx
i_tx_data[63:0] D0 D1 D2 D2 D3 D4 D5
i_tx_empty[5:0] 5
i_tx_startofpacket
i_tx_endofpacket
i_tx_valid
0_tx_ready

The figure above shows how to transmit data using the TX MAC. Data valid
(i_sl_tx_valid) must be held high from the start to end of a packet, unless ready
(o_tx_ready) is low. Valid can be low outside of a packet boundary. A packet FIFO
between the MACsec and E/F-tile TX path takes care of this.

2.2.1.2. Data Transmitting from E/F-tile Hard IP to MACsec

The E/F-Tile Hard IP’s MAC receives data from PHY and transfers it over the AVST
interface. The AVST to AXI-ST conversion logic converts RX MAC data to AXI-ST before
giving it to MACsec IP.

Figure 8. Timing Diagram for Packet Transmitting from E/F-Tile Hard IP to MACsec
i_clk_rx
o_rx_data[63:0] D0 D1 D2 D3 D4 D5
o_rx_empty[5:0] 5
o_rx_startofpacket
o_rx_endofpacket
o_rx_valid

The figure above shows RX MAC client interface receives data.

2.2.1.3. Order of Ethernet Transmission

The TX MAC transmits bytes on the Ethernet link starting with the preamble and
ending with the FCS in accordance with the IEEE 802.3 standard. On the transmit
client interface, the IP core expects the client to send the most significant bytes of the
frame first, and to send each byte in big-endian format. Similarly, on the receive client
interface, the IP core sends the client the most significant bytes of the frame first, and
orders each byte in big-endian format. Whereas the MACsec IP sends and expects
frames to be in least significant bytes first and each byte in little-endian format. The
AVST-AXI-ST bridge handles this ordering between the MACsec and E/F-tile IP.

Send Feedback MACsec Intel FPGA System Design User Guide

11
2. Architecture
767516 | 2023.03.03

Figure 9. Ethernet Frame Octets

Destination Address (DA) Source Address (SA) Type Length (TL) Data (D)
0 c tet 5 4 3 2 1 0 5 4 3 2 1 0 1 0 0 ... NN
B it [47:40] [39:32] [31:24] [23:16] [15:8] [7:0] [47:40] [39:32] [31:24] [23:16] [15:8] [7:0] [15:8] [7:0] M S B [7:0] ... L S B [7:0]

For example, the destination MAC address includes the following six octets AC-
DE-48-00-00-80. The first octet transmitted (octet 0 of the MAC address described in
the 802.3 standard) is AC and the last octet transmitted (octet 7 of the MAC address)
is 80. The first bit transmitted is the low-order bit of AC, a zero. The last bit
transmitted is the high order bit of 80, a one. The following figure shows that in this
example, 0xAC(DA5) is driven on [511:504] in case of AVST and on[7:0] in case of
AXI-ST and 0x80(DA0)is driven on [471:464] in case of AVST and on [47:40] in case
of AXI-ST.

Figure 10. E/F-tile AVST Interface for 100G

i_clk_tx
i_tx_data[511:504] DA5 D50 D114 DA5 D50 D242 DA5 D50
i_tx_data[503:496] DA4 D51 D115 DA4 D51 D243 DA4 D51
i_tx_data[495:488] DA3 D52 D116 DA3 D52 D244 DA3 D52
i_tx_data[487:480] DA2 D53 D117 DA2 D53 D245 DA2 D53
i_tx_data[479:472] DA1 D54 D118 DA1 D54 D246 DA1 D54
i_tx_data[471:464] DA0 D55 D119 DA0 D55 D247 DA0 D55
i_tx_data[463:456] SA5 D56 D120 SA5 D56 D248 SA5 D56
i_tx_data[455:448] SA4 D57 D121 SA4 D57 D249 SA4 D57
i_tx_data[447:440] SA3 D58 D122 SA3 D58 D250 SA3 D58
i_tx_data[439:432] SA2 D59 SA2 D59 D251 SA2 D59
i_tx_data[431:424] SA1 D60 SA1 D60 D252 SA1 D60
i_tx_data[423:416] SA0 D61 SA0 D61 D253 SA0 D61
i_tx_data[415:408] TL1 D62 TL1 D62 D254 TL1 D62
i_tx_data[407:400] TL0 D63 TL0 D63 D255 TL0 D63
i_tx_data[399:392] D0 D64 D0 D64 D0 D64
i_tx_data[391:384] D1 D65 D1 D65 D1 D65
••
•
i_tx_data[23:16] D47 D111 D47 D111 D47 D111
i_tx_data[15:8] D48 D112 D48 D112 D48 D112
i_tx_data[7:0] D49 D113 D49 D113 D49 D113
i_tx_startofpacket
i_tx_endofpacket
i_tx_empty[5:0] 55 50

MACsec Intel FPGA System Design User Guide Send Feedback

12
2. Architecture
767516 | 2023.03.03

Figure 11. MACsec AXI-ST Interface for 100G

2.2.1.4. E/F-Tile Hard IP Reset Sequence

Figure 12. Waveform of the Hard IP Reset Sequence

clk
i_csr_rst_n
i_tx_rst_n
i_rx_rst_n
i_tx_pll_locked
o_tx_lanes_stable
o_rx_block_lock
io_rx_pcs_ready

From the figure above, we see that there are several steps for IP reset.
1. The i_csr_rst_n reset returns all Ethernet registers to their original values,
including the statistics counters.
2. The assertion of i_tx_pll_locked leads to desertion of the i_csr_rst_n output signal.
3. Once i_csr_rst_n deserted, its leads to assertion of o_tx_lane_stable output
signal.
4. After deassertion of i_csr_rst_n reset and once PHY is ready to receive data it
asserts o_rx_block_lock and io_rx_pcs_ready output signal.
5. Asserting the i_csr_rst_n reset leads to desertion of i_tx_lane_stable,
o_rx_block_lock and rx_pcs_ready output signal.

Send Feedback MACsec Intel FPGA System Design User Guide

13
2. Architecture
767516 | 2023.03.03

2.3. Data Path Between MACsec and MCDMA

The datapath at MCDMA interface is AVST with an additional channel ID signal which
differentiates between the PF/VF groups. Your logic should have a mapping table and
routing logic to MUX and DeMUX the packets. This can be done in either AVST or AXI-
ST (post bridge logic) interfaces. The solution’s ecosystem supports an AXI-St MUX/
DeMUX block which you can leverage. Map the AVST Channel ID as AXI-St TID if ID is
supported.

Figure 13. MACSec(s) to MCDMA Path

CDC
Multi Packet F
Seg to Packet Drop/ I
Single Parser F
Seg Ctrl O
MAC
Sec0 AXI-ST
F
Rate Pkt FIFO I
Ctrl F
O
AXI-ST MC
MUX/ AXI-ST to AVST DMA
DeMUX AVST Bride
Multi Packet F
Seg to Packet Drop/ I
Single Parser F
Seg Ctrl O
MAC AXI-ST
Sec1 F
Rate Pkt FIFO I
Ctrl F
O

Figure 14. Timing Diagram for Packet Boundary Alignment (Pkt FIFO in AVST)
clk
tvalid
sop
eop/tlast
tready
tdata D0 D1 D2 D3 D4

tvalid
sop
eop/tlast
tready
tdata D0 D1 D2 D3 D4

The bridge logic shown in Figure 13 above between the MACsec and the MCDMA
supports a few functions as mentioned below:

MACsec Intel FPGA System Design User Guide Send Feedback

14
2. Architecture
767516 | 2023.03.03

• The MACsec AXI-St interface supports multi-packet mode where a packet can start
at a next segment when the current packet ends. But the MCDMA’s AVST interface
treats the entire bus as single segment.Therefore, there is conversion logic which
converts multi-packet to single packet in AXI-St first. This conversion also eases
the packet filtering logic at a fixed bit fields during the start of a packet.
• This path includes CDC FIFOs (DC FIFO) in both the directions to accommodate
the clock domain crossing between MACsec 400MHz and MCDMA 250/500 MHz.
This interface is 512-bits wide.
• The MACsec expects AXI-St valid not be de-asserted in between a packet’s SOP
and EOP unless Ready is deasserted. Therefore, a complete packet must be
available in the FIFO before asserting Valid on the MACsec AXI-St RX interface of
an uncontrolled port using a packet FIFO.
• Handling the packets from the MACsec TX uncontrolled port to MCDMA path as
handshake is not supported. Therefore, you need to either drop the packet or
maintain a bigger buffer. Consider a scenario where the packet is in flight and the
buffer gets filled. To avoid such issues, write a packet into a buffer only when the
empty space in buffer is higher than the MTU (Ethernet packet size). Assume
MCDMA is already prepopulated with a descriptor for D2H Transfer.
• It is understood that MACsec forwards all its received packets on decryption line to
uncontrolled RX interface, without any type of filtering. The encrypted packets are
also routed in this manner. Your logic should implement packet filtering based on
the required Ether type. An example would be dropping every packet when Ether
type is not 0x888E(EAPOL).

Certain applications might require Gen4x8 instead of the Gen4x16 PCIe configuration.
For these cases there can be an adaptor block that converts 256-bit user interface to
512-bit user interface to retain the logic which targets the Gen4x16 PCIe MCDMA.

2.3.1. AXI-ST Multi-Segment to Single-Segment Conversion

Each MACsec’s RX uncontrolled port dynamically gives data in multi-segment format
(where a new packet can start in the same cycle that the current packet ends in)
whenever the data is available to be combined from the same port. The packet parser/
filtering logic works in single segment format and looks for an Ethernet type field at a
fixed location within the packet after SA+DA (Source, Destination MAC Addresses).
The uncontrolled port receives plaintext as well as ciphertext packets too so there
should not be any back pressure from your logic while converting multi-segment to
single segment to maintain the equal incoming and outgoing rate. The uncontrolled
port does not support TREADY de-assertion.

Send Feedback MACsec Intel FPGA System Design User Guide

15
2. Architecture
767516 | 2023.03.03

Figure 15. AXI-ST Multi-Segment Stream Data from Uncontrolled Port

clk
reset
axis_tdata[255:0] AAAA_AAAA BBBB_BBBB 0000_00CC DDDD_DDDD AAAA_AAAA CCCC_00BB EEEE_EEEE EEEEE_EEEE
axis_tdata[511:256] AAAA_AAAA CCCC_CCCC DD00_0000 00EE_EEEE BBBB_AAAA DDDD_CCCC EEEE_EEEE EEEE_EEEE
axis_tvalid
axis_tready
axis_tkeep[31:0] all Fs all Fs 0000_00FF all Fs all Fs FFFF_00FF all Fs all Fs
axis_tkeep[63:32] all Fs all Fs FF00_0000 00FF_FFFF all Fs all Fs all Fs all Fs
axis_tlast
axis_tuser.tlast.seg0
axis_tuser.tlast.seg1
axis_tuser.tlast.seg2
axis_tuser.tlast.seg3
axis_tuser.tlast.seg4
axis_tuser.tlast.seg5
axis_tuser.tlast.seg6
axis_tuser.tlast.seg7

As shown below, short packets may introduce an additional cycle to transfer the same
amount of data coming in multi-segment mode. Your implementation should try to
match the incoming rate by minimizing the unrequired throttling.

Figure 16. AXI-ST Single-Segment Stream Data (Converted)

clk
reset
axis_tdata[255:0] AAAA_AAAA BBBB_BBBB 0000_00CC DDDD_DDDD AAAA_AAAA 00BB_BBBB CCCC_CCCC EEEEE_EEEE EEEE_EEEE
axis_tdata[511:256] AAAA_AAAA CCCC_CCCC 0000_0000 EEEE_EEDD 0000_AAAA 0000_0000 EEEE_DDDD EEEE_EEEE 0000_EEEE
axis_tvalid
axis_tready
axis_tkeep[31:0] all Fs all Fs 0000_00FF all Fs all Fs 00FF_FFFF all Fs all Fs all Fs
axis_tkeep[63:32] all Fs all Fs all 0s all Fs 0000_FFFF all 0s all Fs all Fs 0000 FFFF
axis_tlast

2.3.2. MCDMA
The Multi Channel DMA for PCI Express enables you to efficiently transfer data
between the host and device. It supports multiple DMA channels between the host and
device over the underlying PCIe link. A DMA channel consists of an H2D (host to
device) and D2H (device to host) queue pair.

As shown in the figure below, the Multi Channel DMA can be used in a server’s
hardware infrastructure to allow communication between various VM clients and their
FPGA-device based counterparts. The Multi Channel DMA operates on descriptor-based
queues set up by driver software to transfer data between local FPGA and the host.
The Multi Channel DMA ’s control logic reads the queue descriptors and executes
them.

The Multi Channel DMA IP integrates the Intel® PCIe Hard IP and interfaces with the
host Root Complex via the PCIe serial lanes. On the user logic interface, the Avalon-
MM/Avalon-ST interfaces allow the designer easy integration of the multi Channel DMA
IP for PCI Express with other Platform Designer components.

MACsec Intel FPGA System Design User Guide Send Feedback

16
2. Architecture
767516 | 2023.03.03

Figure 17. MCDMA

Host Intel FPGA

Ch. 0 MCDMA
H2D
Queue H2D
VM QCSR
D2H D2H
Queue QCSR

Ch. 1
H2D H2D
Queue QCSR
VM D2H
D2H Virtual QCSR AVMM/ User
Queue Root PCIe AVST Logic
Machine
Complex HIP Port
Manager

Ch. n
H2D H2D
Queue QCSR
VM
D2H D2H
Queue QCSR

The MCDMA engine operates on a software DMA queue to transfer data between the
local FPGA and the host. The elements of each queue are software descriptors that are
written by driver/software. Hardware reads the queue descriptors and executes them.
Data Mover Blocks for both the directions (D2H, H2D) fetch the descriptors and are
responsible for data transfers to or from the system memory locations specified in the
descriptors or from/to the user application in the hardware.

Figure 18. Descriptor Linked-List

4KB Page
1* Link=0 Q_START_ADDR_L/H (from QCSR)
•
•
•
128* Link=1
4KB Page
129 Link=0
•
•
•
256 Link=1
4KB Page
257 Link=0
•
•
•
384 Link=1 •
•
•
•
• 4KB Page
* = Descriptor index always starts from 1 • n-127 Link=0
•
•
•
n Link=1

Send Feedback MACsec Intel FPGA System Design User Guide

17
2. Architecture
767516 | 2023.03.03

A DMA channel to support Multi Channel DMA data movement consists of a pair of the
descriptor queues: one H2D descriptor queue and one D2H descriptor queue. As
shown in the figure above, the descriptors are arranged contiguously within a 4 KB
page. Each descriptor is 32 bytes in size. The descriptors are kept in host memory in a
linked-list of 4 KB pages. For a 32 byte descriptor and a 4 KB page, each page
contains up to 128 descriptors. The last descriptor in a 4 KB page must be a “link
descriptor” – a descriptor containing a link to the next 4 KB page with the link bit set
to 1. The last entry in the linked list must be a link pointing to the base address
programmed in the QCSR, in order to achieve a circular buffer containing a linked-list
of 4 KB pages.

Software and hardware communicate and manage the descriptors using tail index
pointer (Q_TAIL_POINTER) and head index pointer (Q_HEAD_POINTER) QCSR
registers as shown in the figure below. The DMA starts when software writes the last
valid descriptor index to the Q_TAIL_POINTER register.

Figure 19. Buffer Descriptor (BD) Ring

DESC_IDX
1 Q_HEAD_POINTER
DESC_IDX (Descriptor last fetched by HW)
2
DESC_IDX
3

DESC_IDX
n
Q_TAIL_POINTER
(Last valid descriptor added by SW)

DESC_IDX
1 Q_HEAD_POINTER
DESC_IDX (Descriptor last fetched by HW)
2
DESC_IDX
3

DESC_IDX
n
Q_TAIL_POINTER
(Last valid descriptor added by SW)

MACsec Intel FPGA System Design User Guide Send Feedback

18
2. Architecture
767516 | 2023.03.03

Descriptors between the Q_HEAD_POINTER and Q_TAIL_POINTER are enabled for HW

usage and descriptors between the Q_TAIL_POINTER and Q_HEAD_POINTER are in
SW control. DMA operations for a particular queue pause when both the pointers are
same.

It also offers a DMA-bypass capability to the host for doing PIO Read/Writes to device
memory. This interface is used for downstream logic CSR access. PCIe BAR2 is
mapped to the Avalon-MM PIO Master. Any TLP targeting BAR2 is forwarded to the
user logic. TLP addresses targeting the PIO interface should be 8 bytes aligned. The
PIO interface supports non-bursting 64-bit write and read transfers.

The PIO interface address mapping is as follows: PIO address = {vf_active, pf, vf,
csr_addr}
1. vf_active: This indicates that SRIOV is enabled.
2. pf [PF_NUM-1:0]: Physical function number decoded from the PCIe header
received from the HIP; PF_NUM which is ($clog2(pf_num_tcl)) is the RTL design
parameter selected by you such that Multi Channel DMA IP only allocates required
number of the bits on Avalon-MM side to limit the number of the wires on the user
interface.
3. vf [VF_NUM-1:0]: Virtual function number decoded from the PCIe header received
from the HIP; VF_NUM which is ($clog2(vf_num_tcl)) is the RTL design parameter
selected by you such that Multi Channel DMA IP only allocates required number of
the bits on Avalon-MM side to limit the number of the wires on the user interface.
4. csr_addr [ADDR_SIZE-1:0]: Number of bits required for BAR2 size requested
across all Functions (PFs and VFs) Example: If BAR2 is selected as 4 MB, the
ADDR_SIZE = 22.

The table below shows the DMA channel mapping w.r.t MCDMA IP parameter setting.
• Number of PFs – 2
• Number of VFs per PF – 0
• Number of DMA channels per PF – 1

Table 2. DMA Channel Mapping w.r.t. MCDMA IP Parameters

PF MCDMA SW Channel User Channel

0 0 0

1 0 1

2.4. Data Path Between MACsec and Packet Generator/Checker

(Packet Client)
The path between MACsec and Packet client works in a single clock domain and with
the same data bus width matching the Ethernet data rate. Connections between both
of the modules are 1:1 mapping. Due to potential future Ethernet rate matching for
100G/200G/400G, packet clients are maintained in HW itself and software functions
do not implement data plane termination and sourcing at/from a host. Also MACsec
usecases are not limited to the NIC where the packets are terminated at a TCP port. It
could be a L2, L3 function or a switch.

Send Feedback MACsec Intel FPGA System Design User Guide

19
2. Architecture
767516 | 2023.03.03

2.4.1. Packet Client

The packet client has internal FSMs that generate data based on control from the CSR
interface. Applications running on the host VMs can customize the design’s data traffic
by programming the packet client registers. The host application may also need to
pause/stop the packet transfers by using SA retire. The packet client produces AVST
data streams which needs to be converted to AXI-ST using bridge adaptors.

Below is an example sequence of the CSR access needed to enable the packet client.
Please note that the sequence below does not cover all the available CSR options.
1. Start Packet client Tx by setting CFG_PKT_CL_CTRL[0] to ‘1’ (offset 0x0, value
0x01).
2. Wait for data traffic to complete & counters to update.
3. Set Status Snapshot capture bit by setting CFG_PKT_CL_CTRL[6] to ‘1’ (offset
0x0, value 0x41).
4. Read Status counters (offsets 0x20 to 0x4C) & verify.
5. Clear Status Snapshot capture bit by setting CFG_PKT_CL_CTRL[6] to ‘0’ (offset
0x0, value 0x1).
6. Set CSR Status Clear bit by setting CFG_PKT_CL_CTRL[7] to ‘1’ (offset 0x0, value
0x81).
7. Clear CSR Status Clear bit by setting CFG_PKT_CL_CTRL[7] to ‘0’ (offset 0x0,
value 0x1).
8. Stop Packet client & clear all internal counters (offset 0x0, value 0x100).

2.4.2. Packet Generation and Check

The packet generation block receives settings like packet mode, packet size, IPG,Idle
cycle, packet transfer enable, packet smac and packet dmac address from the packet
client CSR, as shown in the figure below. Based on this information, the internal state
machine creates packets matching the packet size with incremental payload.

MACsec Intel FPGA System Design User Guide Send Feedback

20
2. Architecture
767516 | 2023.03.03

Figure 20. Packet Generator and Checker

Packet Generator
dmac_addr
smac_addr
pkt_start_size AVST AXI-ST
pkt_end_size
Stats
FSM Counters
Packet Client stats
CSR ASVT
--
AXI-ST
csr_clk stats Packet Checker
cfg_tx_pkt_gen AVST AXI-ST
Stats
AVMM Counters

macsec_clk
AXI-Lite to
AXI-Lite AVMM Bridge

This solution enables full duplex data transfers so the patten generation and checking
can be similar in both directions. The packet checker received data is compared with
the reference data from the packet generator. It increments the error count if the
reference data does not match with the received data. The packet checker also
includes statistics counters which count the tx_sop,rx_sop,tx_eop and rx_eop signal
assertions. If the received and transmitted byte count matches the packet size, then
the test done signal gets asserted and produces the packet count on the CSR.

The packet generator supports dynamic packet sizes where subsequent packet sizes
increment in size by a programmed value which is defaulted to 1.

2.5. Data Path Illustrations

The MKA Key exchange path from VM1 to VM0 is shown in the diagram below. The
VM1 prepares MCDMA H2D DMA descriptors whereas VM0 prepares D2H DMA
descriptors in the host memory and initialize the MCDMA CSRs accordingly. When VM1
start an H2D DMA operation, MKA packets are transferred from the host memory to
the FPGA via channel 1 and an MSIX interrupt is triggered to indicate H2D DMA
completion. Received packets are streamed to an uncontrolled port of MACSec-1 after
aligning to the packet boundary (MACsec does not support Idle cycles in between
packets unless tready backpressure happens). These packets are bypassed by the
MACsec IP without any processing, and are transmitted over LAN. Upon receiving
uncontrolled packets at MACsec-0, they are bypassed to its uncontrolled stream
output port without any processing. The MCDMA triggers its D2H DMA on channel 0 to
transfer packets from the FPGA to the host followed by an MSIX interrupt to indicate
D2H DMA completion. The same sequence is followed along VM0 to VM1 (DMA channel
0 to DMA Channel 1) while sending MKA reply packets.

Send Feedback MACsec Intel FPGA System Design User Guide

21
2. Architecture
767516 | 2023.03.03

Figure 21. MKA Exchange Data Traffic

PF1
Port Enable
w.r.t PF/VF

••••
PF0
PF0 •••• PF1
Custom RTL Logic
Quartus IP Packet
Inter- Generator

CSR
connect ICA Interconnect
& Checker HIP
Crypto
AES

AXI-ST
AVMM
Bar AVMM to

AXI-ST
Clock Rate CTL
Bridge Interpreter AXI-Lite
CSR
Controlled Port CSR
PIO Port MUX/
CDC Packet Multi Seg DeMUX

Common Port
F Drop/Ctrl to Single Port AXI-ST P0
I (Packet Seg & MACSec0

Uncontrolled Port
VM1 F Filter) Width Conv MUX/ Pkt
(Server)
O
AXI-ST DeMUX FIFO
F Pkt Rate
I
F FIFO CTL
O QSFP
E/F-Tile Loop-
MCDMA

AXI-St AXI-St
P-Tile

to AVST MUX/
Bridge DeMux F back
I Packet Multi Seg CSR

Uncontrolled Port
F Drop/Ctrl to Single
O (Packet Seg &
VM0 Filter) Width Conv AXI-ST

Common Port
(Key F
I
Port AXI-ST P8
Client) F Pkt Rate
MACSec1 MUX/ Pkt
O DeMUX
FIFO CTL Port MUX/ FIFO
DeMUX
Host Controlled Port
HSSI SS

AXI-ST
SW HW Rate CTL
AXI-ST

Crypto
AES
Packet ICA
Generator HIP
CSR

& Checker

Once a key exchange is done, the host may configure the MACsec IP with the key
information and turn on its packet generator to start transmitting data. Here, since
packets are received at the destination, it is important to have the same packet
generator configuration at both ends in order to transmit and verify the received data.
Once a packet generator is started, it generates the AXI stream packets until a stop
condition is reached. The MACsec encrypts all the received packets using the Crypto
engine and transmits them over a LAN. Upon receiving the packets at the other
MACsec, traffic is decrypted and fed to a packet checker. The checker module
compares the traffic against a reference pattern and updates its status registers. The
application may stop the traffic generator and restart if the system undergoes a
rekeying sequence.

MACsec Intel FPGA System Design User Guide Send Feedback

22
2. Architecture
767516 | 2023.03.03

Figure 22. Packet Generator Data Traffic

Port Enable
w.r.t PF/VF

••••
Custom RTL Logic PF0
Quartus IP Packet
Inter- PF0 •••• PF1
Generator

CSR
connect ICA
& Checker HIP
Crypto Inter-
AES connect

AXI-ST
AVMM
Bar AVMM to

AXI-ST
Clock Rate CTL
Bridge Interpreter AXI-Lite
CSR
Controlled Port CSR
PIO Port MUX/
CDC Packet Multi Seg
DeMUX

Common Port
F Drop/Ctrl to Single Port
I (Packet Seg & AXI-ST P0

Uncontrolled Port
VM1 F Filter) Width Conv MACSec0 MUX/ Pkt
(Server)
O
AXI-ST DeMUX FIFO
F Pkt
I Rate
F FIFO CTL
O E/F-Tile QSFP
Loop-
MCDMA

AXI-St AXI-St
P-Tile

to AVST MUX/
Bridge DeMux F back
I Packet Multi Seg CSR

Uncontrolled Port
F Drop/Ctrl to Single
O (Packet Seg &
VM0 Filter) Width Conv AXI-ST

Common Port
(Key F
I
Port AXI-ST P8
Client) F Pkt Rate
MACSec1 MUX/ Pkt
O DeMUX
FIFO CTL Port MUX/ FIFO
DeMUX
Host Controlled Port
HSSI SS

AXI-ST
SW HW Rate CTL

AXI-ST Crypto
AES
Packet ICA
Generator HIP
CSR

& Checker

2.5.1. MACsec Interface Signal Names

This section gives reference to the MACsec IP interface signal names to avoid any
confusion while debugging the design as it includes many signals and directions of
packet flow.

Figure 23. MACSec IP Interface Signal Names

PKT Client
encrypt_user_ss_p0

decrypt_ss_user_p0

Crypto
app_pp

pp_app

Lorem ipsum
AXI-ST

AXI-ST

Port MUX/ CSR

DeMUX decrypt_user_ss_p0
Controlled Port app_pp
Common Port

Port
dec_byp_ip_app_tx MACSec0 MUX/ AXI-ST ETH
Uncontrolled

DeMUX MAC
AXI-ST encrypt_ss_user_p0
Port

MCDMA
pp_app
enc_byp_app_ip_rx

Lorem ipsum
Send Feedback MACsec Intel FPGA System Design User Guide

23
2. Architecture
767516 | 2023.03.03

2.6. Interrupts
The MCMDA interface supports 4 MSI-X interrupts per channel for one user interrupt
per queue in each direction. If your logic needs to support multiple interrupts, then
you should implement local logic which retains the interrupt requests from different
sources and waits for their turn to request an interrupt to the MCDMA which in turn
generates the MSI-X. A SW routine to read the interrupt source can then take further
action.

Each DMA channel is allocated 4 MSI-X vectors:

• 2’b00: H2D DMA Vector
• 2’b01: H2D Event Interrupt
• 2’b10: D2H DMA Vector
• 2’b11: D2H Event Interrupt

You may use H2D event interrupt or D2H event interrupt to signal pre-defined events.
In the interrupt vector table, these 4 vector entries are available per channel. The SW
is expected to configure the entries if it is expecting an interrupt from your logic.

An interrupt controller as shown in the figure below is implemented. Currently there

are only two MACsec interrupts as irq sources (one per channel) and only one is
outstanding at any given time.

Figure 24. Interrupt Controller

AXI-Lite Interrupt Controller

Interface C
S Status Reg[N-1:0]
R
Enable Reg[N-1:0]
Clear Reg[N-1:0] Valid
IRQN-1 Data
IRQN-2 IR Q Logic & Event Ready
•• Mapping
IRQ 0

AXI-Lite
Channel 0
Interrupt
MACSec 0 IRQ Controller
MSIx Valid
MUX MSIx Data MCDMA
AXI-Lite MSIx Ready
Channel 1
Interrupt
Controller
MACSec 1 IRQ

Each channel’s interrupt controller may support up to N number of IRQ signals from
different modules. The usr_event_msix_data_i that goes to MCDMA indicates the
IRQ from which channel only. The SW Interrupt routine’s responsibility is to probe the

MACsec Intel FPGA System Design User Guide Send Feedback

24
2. Architecture
767516 | 2023.03.03

register space of each interrupt controller to know which input IRQ caused the channel
specific interrupt. Based on this the SW can go and read the interrupt status register
of the IP module that asserted the IRQ. The two-step process from the SW is
mentioned above. The SW application can clear any particular interrupt bit in the
interrupt controller after servicing it and can then service the next one if it is asserted
again.

2.6.1. MCDMA MSI-X Table Configuration

The MCDMA comes with its own memory offset space for MSI-X Table and PBA Table
for each function enabled. It is always tied to the BAR0 address space of each
function. Settings like Table BIR/PBA BIR and Table Offset/PBA Offset are not valid. As
shown below, the MSI-X Table starts at 0x10_0000 offset on each function’s BAR0 with
maximum size allocated of 512KB (but requires only 32KB to support maximum 2048
MSI-X vectors as per spec.). The PBA Table starts at offset of 0x18_0000 on each
function’s BAR0.

Table 3. MCDMA Address Space

Address Space Name Range Size Description

QCSR (D2H, H2D) 22'h00_0000 - 22'h0F_FFFF 1MB Individual queue control

registers. Up to 2048 D2H
and 2048 H2D queues.

MSI-X (Table and PBA) 22'h10_0000 - 22'h1F_FFFF 1MB MSI-X Table and PBA space.

GCSR 22'h20_0000 - 22'h2F_FFFF 1MB General DMA control and

status registers.

Reserved 22'h30_0000 - 22'h3F_FFFF 1MB Reserved.

The current solution supports only 4 MSI-X vectors per PF, out of it 2 are dedicated for
the MCDMA internal use. The table below gives the exact offsets for each usage per
PF.

Table 4. Address Offsets Per PF

Address Offset Usage Description

BAR0 + 0x10_0000 + 0x00 H2D DMA Vector DMA Internal Use for H2D descriptor
updates

BAR0 + 0x10_0000 + 0x10 H2D Event Interrupt Reserved

BAR0 + 0x10_0000 + 0x20 H2D DMA Vector DMA Internal Use for D2H descriptor
updates

BAR0 + 0x10_0000 + 0x30 H2D Event Interrupt MACSec Interrupt

2.7. Packet FIFO

The Ethernet MAC (TX) as well as the MACsec IP’s uncontrolled ports expect data at a
packet boundary and may be unpredictable if the valid goes low in between a packet
transfer after the start of the packet and before end of the packet. This can be
implemented either on the AVST or AXI streaming interfaces. The figure below shows
the implementation for the Avalon streaming (AVST) interface. Every EOP on the input
side increments the tracking counter on the output side. Writing data into FIFO
happens on a clock cycle boundary, but the reading from FIFO is controlled at the
packet boundary. The read enable logic on FIFO read asserts only when the packet
counter is non-zero. The packet counter increments by 1 when the EOP is read from

Send Feedback MACsec Intel FPGA System Design User Guide

25
2. Architecture
767516 | 2023.03.03

the FIFO. The packet counter value does not change when both the increment and
decrement flags get asserted in the same clock cycle. The packet FIFO works in a
single clock domain.

Figure 25. Single Clock Domain Packet FIFO

SOP EOP Empty Data

valid wren delay valid

ready ~fifo full rden ready
AVST data data AVST
Input empty empty Output

cnt ≠ 0
sop sop
eop ~fifo empty eop
SC FIFO

Increment cnt Packet Counter Decrement cnt

2.8. AXI-ST Rate Controller

The MACsec IP does not support back pressure in the current implementation. For
example, If the MACsec is being configured to work at 25G rate, then it is your
responsibility to control the incoming traffic from different interfaces (uncontrolled/
controlled ports) to ~70% of the rate defined to accommodate the SecTAG Header
which the MACsec IP inserts into the data traffic which is a significant overhead during
lower packet sizes. Logic coordinates between the two interfaces (uncontrolled,
controlled ports) and can decide weights for each interface. This is implemented with
parameters during compile time. This logic de-asserts the tready to different sources
of traffic i.e. from MCDMA to uncontrolled port (MKA traffic) and from packet
generator to controlled port.

The design to control the aggregated input rate to accommodate the overheads
inserted by MACSec which targets the actual line rate is shown in the figure below.
There are multiple ways of doing this.

As mentioned above, there is a need to distribute the total bandwidth statically

between two interfaces of MACSec i.e. control path (uncontrolled port) and data path
(controlled port) where the priority can be given to data path as in Deficit Weighted
Round Robin (DWRR) scheduler.

MACsec Intel FPGA System Design User Guide Send Feedback

26
2. Architecture
767516 | 2023.03.03

Figure 26. Rate Controller Module (Highlighted in Red)

P-Tile HW
Host Interface
Interrupt
Control MCDMA PIO

Channel Decoder (CH0/CH1-PFO, CH2/CH3-PF1)

AXI-AVST
CDC FIFOs
Pkt FIFO
tready Pkt Filter * EAPOL
5 Gbps 12 Gbps
Rate Cnt Multi-seg con v
E-Tile tready=1
Uncontrolled Port CSR CSR

Controlled Port
Rate Cnt
Common Port
AXI-AVST

TX Pkt FIFO
tready=1 tready Packet Gen
AXI-ST Pkt Filter
RX
MACSec0 & Chk
Crypto ICA
25 Gbps AES HIP

If the MACsec IP expects the controlled rate at every packet level, then the number of
idle cycles to be inserted by rate controller depends on the number of valid data cycles
received. Granularity of these calculations are restricted to clock cycle for ease of
timing, i.e. it assumes that all the data bytes are valid in EOP (8 Byte granularity on a
64 bit bus). The design assumes the maximum possible data rate as “Bus width x
Clock frequency”, i.e. for a 25G MACSec, controlled port data width is 64b running at
400 MHz which results in max rate of 25.6 Gbps (MACSEC_MAX_RATE) passed to
module as a parameter. The number of idle cycles required after end of the current
packet depends on another parameter (MACSEC_MAX_IN_RATE) that defines the
expected input rate. For example, if the expected input rate is 12 Gbps then the
number of idle cycles for each valid clock cycle in a packet is defined by a local
parameter as (MACSEC_MAX_RATE/MACSEC_MAX_IN_RATE). The design maintains a
counter that increments by this ratio on every valid cycle between start of packet and
end of packet. The same counter decrements after EOP cycle until it reaches 0. During
this time, the AXI-ST TREADY is de-asserted.

Figure 27. Null Cycles Inserted After Every Packet Transfer

4 Cycles 8 Cycles •• 2 Cycles 4 Cycles

Valid Data Null Data Valid Data Null Data

Time

2.9. Error Handling

Errors are possible at different levels in any system and system level handling of a few
errors is always important as some error scenarios may be fatal, non-fatal and not
correctable. Even if they are correctable, the system level application should
understand the performance impact due to this. In this system level example design,
there are 3 main interfaces i.e. Ethernet MAC, MACsec and PCIe where the potential
errors are possible.

Send Feedback MACsec Intel FPGA System Design User Guide

27
2. Architecture
767516 | 2023.03.03

2.10. Top Level Signals

The table below lists the top level signals.

Table 5. Top Level Signals

Signal Direction Width Description

fpga_clk_100 Input 1 Clock input for CSR access

fpga_clk_156p25 Input 1 156.25 MHz for Ethernet

HSSI refclk

fpga_clk_100_pcie Input 1 100 MHz for PCIe refclk

fpga_reset_n Input 1 External reset input

(pin_perst_n)

Ethernet Interface

p0_rx_serial Input 2 Transceiver phy serial input

data

p0_tx_serial Output 2 Transceiver phy serial output

data

p8_rx_serial Input 2 Transceiver phy serial input

data

p8_tx_serial Output 2 Transceiver phy serial output

data

qsfp_modsel Output 2 QSFP mode selection

qsfp_lowpwr Output 2 QSFP low power signal

qsfp_rstn Output 2 QSFP reset pin

PCIe Interface

pcie_p0_rx_serial Input 16/8 PCIe Transceiver phy serial

input data

pcie_p0_tx_serial Output 16/8 PCIe Transceiver phy serial

output data

MACsec Intel FPGA System Design User Guide Send Feedback

28
767516 | 2023.03.03

Send Feedback

3. Interface Overview
This section covers the interface overview.

3.1. Clocking
This design uses multiple clock domains as the Ethernet MAC (25G/100G) works at a
different rate compared to the PCIe+MCDMA (128G) and the MACsec (200G). The
interface clocks are shown below.

Table 6. Clocking Interfaces

User Interface Clock Frequency (MHz) Remarks

HSSI-SS AXI-ST Interface 402.832 Fixed for 25G/100G configuration

HSSI-SS AXI-Lite Interface 100 CSR clock. Use a bridge for MCDMA
app_clk to 100

MACSec AXI-ST interface 400 As per MACSec HAS requirement

(throughput of 200G)

MACSec AXI-Lite Interface 100 CSR clock

Crypto AXI-ST Interface 400 As per Crypto solution HAS for inline
processing

MCDMA AVST Interface 250/500 For Gen3x16/Gen4x16 configuration

respectively

Packet Generator/Checker 400 MACSec interface clock

Intel Corporation. All rights reserved. Intel, the Intel logo, and other Intel marks are trademarks of Intel
Corporation or its subsidiaries. Intel warrants performance of its FPGA and semiconductor products to current
specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any ISO
products and services at any time without notice. Intel assumes no responsibility or liability arising out of the 9001:2015
application or use of any information, product, or service described herein except as expressly agreed to in Registered
writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying
on any published information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
3. Interface Overview
767516 | 2023.03.03

Figure 28. Clocking Structure

156.25 MHz
400 MHz
100 MHz
400 MHz
100 MHz Interconnect
402.832 MHz ICA
600 MHz HIP
Packet
AVMM BAR Generator Cypto
Clock Interpreter AVMM to & Checker AES

CSR
Bridge AXI-Lite

CDC
CDC CSR
PIO Multi
F Packet
I Seg to
F Drop/ Single Port
O Ctrl Seg CSR MACSec0 MUX/
F
DeMUX
I Pkt Rate i_macsec_clk P0
AXI-St F FIFO CTL
to AXI-St O
AVST MUX/ o_p0_pll_clk
MCDMA

Bride DeMUX F Packet Multi

E/F-Tile
P-Tile

I Seg to o_p8_pll_clk
F Drop/ Single
O Ctrl Seg
P8
F CSR Port
I Pkt Rate
F FIFO CTL MACSec1 MUX/
O
i_macsec_clk DeMUX

i_ref_clk[0] HSSI SS

Packet Cypto
External Generator AES External
IO PLL
CSR

PLL & Checker ICA PLL

HIP

Table 7. Reference Clock Frequencies

Reference Clocks Clock Frequency (MHz) Remarks

HSSI-SS or E/F-Tile 156.25 External source

PCIe 100 External source

IOPLL 100 External source. Generate MACSec,

Crypto and CSR clock sources

3.2. Resets
The system reset for PCI Express (pin_perst_n) driven by the downstream port
through the edge connector is the only hard reset available to the entire design. This
reset is provided as an output by the endpoint core which is connected to the MCDMA.

In addition, a software driver programmed reset (soft rest), called app_rst_n, is

provided by the MCDMA (SW_RESET register[0]) which resets MCDMA soft IP blocks
as well as the user logic. This is allowed to be driven by PF0 only. The soft reset
feature is used when a driver is loading and unloading to flush out all pending
transactions and prevent any stale transactions from interrupting the host when the
hardware is not in use.

MACsec Intel FPGA System Design User Guide Send Feedback

30
3. Interface Overview
767516 | 2023.03.03

Figure 29. Reset Tree

Custom RTL Logic
Quartus IP
ICA
Interconnect Packet HIP
Generator Crypto
& Checker AES
AVMM
Clock Bar AVMM to
Bridge Interpreter AXI-Lite aes_ip_app_rst_n aes_app_ip_rst_ack_n

CSR CSR
PIO Multi
subsystem_cold_
F
I Packet Seg to rst_n
F Drop/ Single
O Ctrl Seg
rst- MACSec0 P0
F Hand- Single ctrl
I shake Seg to subsystem_cold_
F at pkt Multi rst_n
AXI-St AXI-St O boundary Seg subsystem_cold_
to AVST MUX/ rst_ack_n
Bridge DeMux rst-
MCDMA

F Multi
E/F-Tile
P-Tile

Packet Seg to ctrl

I Drop/ Single
F
O Ctrl Seg CSR
subsystem_cold_
Hand- Single rst_n subsystem_cold_
F
I shake Seg to rst_ack_n
F at pkt
boundary
Multi
Seg rst- MACSec1 P8
O
ctrl

subsystem_cold_ HSSI SS
reset_status_n app_rst_n rst_ack_n aes_ip_app_rst_n
aes_app_ip_rst_ack_n
Packet Crypto
Generator AES
& Checker ICA
HIP

Send Feedback MACsec Intel FPGA System Design User Guide

31
767516 | 2023.03.03

Send Feedback

4. Parameters
This table below lists the parameters.

Table 8.
Name Default Value Description

PCIE_LANE_W 16 Number of PCIe lanes; x16 or x8.

NUM_MACSEC_INST 2 Number of MACsec blocks in the

design.

MACSEC_CSR_ADDR_W 25 32MB CSR address space by default.

MACSEC_CSR_DATA_W 64 64 bit CSR data path with unaligned 32

bit support too.

PKTCLI_CSR_ADDR_W 12 4K CSR address space for packet

client.

PKTCLI_CSR_DATA_W 32 32 bit CSR data path for packet client.

NUM_MAC_CHANNELS 2 Matches with NUM_MACSEC_INST as

MAC works with MACsec.

NUM_LANES 1 Number of QSFP lanes. For 25G its 1;

for 100G its 4.

TILE_DATA_WIDTH 64 MAC AVST data width. For 25G it is 64;

for 100G it is 512.

TILE_EMPTY_WIDTH 3 MAC AVST empty width. For 25G it is

3; for 100G it is 6.

Intel Corporation. All rights reserved. Intel, the Intel logo, and other Intel marks are trademarks of Intel
Corporation or its subsidiaries. Intel warrants performance of its FPGA and semiconductor products to current
specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any ISO
products and services at any time without notice. Intel assumes no responsibility or liability arising out of the 9001:2015
application or use of any information, product, or service described herein except as expressly agreed to in Registered
writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying
on any published information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
767516 | 2023.03.03

Send Feedback

5. Configuration Registers
This section covers configuration registers.

5.1. System Register Map

Refer to the respective IP/subsystem documentation for internal address offsets. At
the system level, BAR2 of each PF drive the downstream CSR space through the
MCDMA PIO interface. Registers inside MCDMA are targeted to BAR0 of each PF for
Queue control (D2H and H2D directions) and MSI-X table updates. General CSR
(GCSR) of MCDMA is only maintained by PF0-BAR0.

The PIO address is arranged as {vf_active,clog2(PF_NUM),clog2(VF_NUM),PIO BAR2

Address} where vf_active, clog2(PF_NUM) and clog2(VF_NUM) are declared as 1-bit
each since we support only 2 PFs and no VFs. Each PF’s BAR2 can be enabled for
32MB i.e. 26-bit interface and the upper address bits differentiate the addressing
between 2 PFs. The PIO interface is accessible irrespective of configuring the datapath
queues. Since this design does not support any VFs, vf address bits are removed from
AVMM address using BAR interpreter logic which reduces the needed address range for
design.

This path supports 64-bit register access as well as 32-bit register access with
adapters in the path. Port MUX/DeMUX/Crypto interfaces all are 32-bit accesses. The
current MACsec IP does not support these interfaces but it is under evaluation to
expose them in the future.

Intel Corporation. All rights reserved. Intel, the Intel logo, and other Intel marks are trademarks of Intel
Corporation or its subsidiaries. Intel warrants performance of its FPGA and semiconductor products to current
specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any ISO
products and services at any time without notice. Intel assumes no responsibility or liability arising out of the 9001:2015
application or use of any information, product, or service described herein except as expressly agreed to in Registered
writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying
on any published information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
5. Configuration Registers
767516 | 2023.03.03

Figure 30. Address Map from MCDMA PIO Interface

AVMM- MM 0x0000_0000 – 0x01FF_FFFF
BAR2 PIO Interconnect MACSec0
32 MBytes
BAR0
Packet 0x0200_0000 – 0x0200_0FFF
MCDMA Client0 4 kBytes
PCIe
0x0200_1000 – 0x0200_12FF
IRQ CTL0 256 Bytes

Ethernet 0x0200_2000 – 0x0200_2FFF

MAC0 4 kBytes

0x0200_3000 – 0x0200_6FFF
Port MUX0

Based on MACSec IP Support

16 kBytes

Port 0x0200_7000 – 0x0200_AFFF

DeMUX0 16 kBytes

0x0200_B000 – 0x0200_B1FF
ICA0
512 kBytes

0x0400_0000 – 0x0500_FFFF
MACSec1 32 MBytes

Packet 0x0600_0000 – 0x0600_0FFF

Client1 4 kBytes

0x0600_1000 – 0x0600_12FF
IRQ CTL1 256 Bytes

Ethernet 0x0600_2000 – 0x0600_2FFF

MAC1 4 kBytes

0x0600_3000 – 0x0600_6FFF
Port MUX1
16 kBytes
Based on MACSec IP Support

Port 0x0600_7000 – 0x0600_AFFF

DeMUX1 16 kBytes

0x0600_B000 – 0x0600_B1FF
ICA0
512 Bytes

5.1.1. Packet Client Register Map

The table below describes the packet client CSRs.

MACsec Intel FPGA System Design User Guide Send Feedback

34
5. Configuration Registers
767516 | 2023.03.03

Table 9. Packet Client Register Map

Offset Name Bit Type HW Reset Value Description

0x000 CFG_PKT_CL_CTR [31:20] RO 23’b0 Reserved.

L
[19:12] RW 8’b0 Number of Idle
cycles to be
inserted between
packets.
Applicable in
dynamic mode
when
CFG_PKT_CL_CTR
L[9] is set to 1.

[11:10] RW 2’b00 00: Reserved

(Random).
01: Fixed length
mode.
10: Incremental
length mode.
11: Reserved.
Only applicable in
dynamic packet
gen mode.

[9] RW 1’b0 0: For

transmission with
random gap in
between packets.
Only applicable in
Dynamic packet
gen mode.
1: For
transmission with
fixed gap in
between packets.

[8] RW 1’b0 When set to ‘1’, it

clears Internal Tx
& Rx Status
counters. Once
set to ‘1’, CPU
must clear this bit
afterwards.

[7] RW 1’b0 When set to ‘1’, it

clears Tx & Rx
Status counter
CSRs. Once set to
‘1’, CPU must
clear this bit
afterwards.

[6] RW 1’b0 When set to ‘1’, it

takes snapshot of
Internal Tx & Rx
Status counters to
CSRs. Once set to
‘1’, CPU must
clear this bit.

[5] RW 1’b0 0: Disable Packet

checker
.
1: Enable Packet
checker.
continued...

Send Feedback MACsec Intel FPGA System Design User Guide

35
5. Configuration Registers
767516 | 2023.03.03

Offset Name Bit Type HW Reset Value Description

Checker block
generates
reference data
based on its own
packet generator
setting.

[4] RW 1’b0 0: Reserved.

1: Select dynamic
mode packet
generation.

[3] RW 1’b0 0: Deassert

checker/
packetgen soft
reset.
1: Assert checker/
packetgen soft
reset
CSR block is not
reset by this soft
reset. Use this to
soft reset blocks
after a sequence
execution.

[2] RW 1’b0 0: Generates

traffic in one shot
mode.
1: Reserved.

[1] RO 1’b0 Reserved.

[0] RW 1’b0 1: Tx Traffic is

enabled from
packet client.
0: Tx traffic is
disabled from
packet client.

0x004 RESERVED [31:16]

[15:0]

0x008 RESERVED [31:16]

[15:0]

0x00C DYN_DMAC_ADDR [31:16] RO 0x0 Reserved.

_U
[15:0] RW 0x1234 Destination MAC
address (upper 16
bits).
Only applicable in
dynamic packet
mode.

0x010 DYN_DMAC_ADD [31:0] RW 0x56780A Destination MAC

DR_L DD address (lower 32
bits).
Only applicable in
dynamic packet
gen mode.

0x014 DYN_SMAC_ADDR [31:16] RO 0x0 Reserved.

_U
continued...

MACsec Intel FPGA System Design User Guide Send Feedback

36
5. Configuration Registers
767516 | 2023.03.03

Offset Name Bit Type HW Reset Value Description

[15:0] RW 0x8765 Source MAC

address (upper 16
bits).
Only applicable in
dynamic packet
gen mode.

0x018 DYN_SMAC_ADDR [31:0] RW 0x0x4321 Source MAC

_L 0ADD address (lower 32
bits).
Only applicable in
dynamic packet
mode.

0x01C DYN_PKT_NUM [31:0] RW 0xA Specifies the

number of
packets to
transmit from the
packet generator.
Only applicable in
dynamic packet
gen mode.

0x020 DYN_PKT_SIZE_C [31:30] RO 0x0

FG
[29:16] RW 0x2580 Specifies the
transmit packet
size in bytes.
Specifies the
upper limit of the
packet size in
bytes. This is only
applicable to
incremental mode
within dynamic
gen packet mode.

[15:14] RO 0x0 Reserved.

[13:0] RW 0x0040 Specifies the

transmit packet
size in bytes.
For fixed mode,
these bits specify
the transmit
packet size in
bytes.
For incremental
mode, these bits
specify the
incremental bytes
for a packet.
Only applicable in
dynamic packet
gen mode.

0x024 STAT_TX_SOP_CN [31:0] RO 32’b0 TX Start of packet

T_L counter lower
32bits. CPU may
clear this CSR by
setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x028 STAT_TX_SOP_CN [31:0] RO 32’b0 TX Start of packet

T_U counter upper
32bits. CPU may
continued...

Send Feedback MACsec Intel FPGA System Design User Guide

37
5. Configuration Registers
767516 | 2023.03.03

Offset Name Bit Type HW Reset Value Description

clear this CSR by

setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x02C STAT_TX_SOP_CN [31:0] RO 32’b0 TX End of packet

T_L counter lower
32bits. CPU may
clear this CSR by
setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x030 STAT_TX_SOP_CN [31:0] RO 32’b0 TX End of packet

T_U counter upper
32bits. CPU may
clear this CSR by
setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x034 STAT_TX_ERR_CN [31:0] RO 32’b0 TX Error counter

T_L lower 32bits. CPU
may clear this
CSR by setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x038 STAT_TX_ERR_CN [31:0] RO 32’b0 TX Error counter

T_U upper 32bits. CPU
may clear this
CSR by setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x03C STAT_RX_SOP_CN [31:0] RO 32’b0 RX Start of packet

T_L counter lower
32bits. CPU may
clear this CSR by
setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x040 STAT_RX_SOP_CN [31:0] RO 32’b0 RX Start of packet

T_U counter upper
32bits. CPU may
clear this CSR by
setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x044 STAT_RX_EOP_CN [31:0] RO 32’b0 RX End of packet

T_L counter lower
32bits. CPU may
clear this CSR by
setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x048 STAT_RX_EOP_CN [31:0] RO 32’b0 RX End of packet

T_U counter upper
32bits. CPU may
clear this CSR by
setting
CFG_PKT_CL_CTR
L[7] to ‘1’.
continued...

MACsec Intel FPGA System Design User Guide Send Feedback

38
5. Configuration Registers
767516 | 2023.03.03

Offset Name Bit Type HW Reset Value Description

0x04C STAT_RX_ERR_CN [31:0] RO 32’b0 RX Error counter

T_L lower 32bits. CPU
may clear this
CSR by setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x050 STAT_RX_ERR_CN [31:0] RO 32’b0 RX Error counter

T_U upper 32bits CPU
may clear this
CSR by setting
CFG_PKT_CL_CTR
L[7] to ‘1’.

0x054 STAT_SYSTEM_MI [31:5] RO 31’b0 Reserved.

SC
[4] RO 1’b0 0: HSSI SS
rx_pcs_ready is
not asserted.
1: HSSI SS
rx_pcs_ready is
asserted.

[3] RO 1’b0 0: HSSI SS

tx_pll_locked is
not asserted.
1: HSSI SS
tx_pll_locked is
asserted.

[2] RO 1’b0 0: HSSI SS

tx_lanes_stable is
not asserted.
1: HSSI SS
tx_lanes_stable is
asserted.

[1] RO 1’b0 0: System Reset

Sequence is not
complete.
1: System Reset
Sequence is
complete.

[0] RO 1’b0 0: SADB

Configuration is
not complete.
1: SADB
Configuration is
complete.

0x058 STAT_CHECKER_ [31:1] RO 28’b0 Reserved.

MISC
[1] RO 1’b0 Reserved.

[0] RO 1’b0 1: Data

mismatches at
data checker.
0: Data does not
mismatch.

0x05C STAT_CHECKER_C [31:1] RO 32’b0 Live Packet count

NT received at the
checker.
continued...

Send Feedback MACsec Intel FPGA System Design User Guide

39
5. Configuration Registers
767516 | 2023.03.03

Offset Name Bit Type HW Reset Value Description

0x060 PKTCLI_RX_BYTE [31:0] RO 32’h0 Lower bits for

_CNT_L counter to count
number of bytes
received.

0x064 PKTCLI_RX_BYTE [31:0] RO 32’h0 Upper bits for

_CNT_U counter to count
number of bytes
received.

0x068 PKTCLI_TX_BYTE_ [31:0] RO 32’h0 Lower bits for

CNT_L counter to count
number of bytes
sent.

0x06C PKTCLI_TX_BYTE_ [31:0] RO 32’h0 Upper bits for

CNT_U counter to count
number of bytes
sent.

0x070 PKTCLI_TX_NUM_ [31:0] RO 32’h0 Number of clock

TICKS_L cycles for all the
packets to be
sent. (lower bits).

0x074 PKTCLI_TX_NUM_ [31:0] RO 32’h0 Number of clock

TICKS_U cycles for all the
packets to be
sent. (upper bits).

0x078 PKTCLI_RX_NUM_ [31:0] RO 32’h0 Number of clock

TICKS_L cycles for all the
packets to be
received. (lower
bits).

0x07C PKTCLI_RX_NUM_ [31:0] RO 32’h0 Number of clock

TICKS_U cycles for all the
packets to be
received. (upper
bits).

5.1.2. Interrupt Controller Register Map

The table below describes the interrupt controller register map.

Table 10. Interrupt Controller Register Map

Offset Name Bit Type HW Reset Value Description

0x000 INTC_CSR_STATU [31:1] RO 00 Reserved.

S
[0] RO 00 To check the
status of the
interrupt.
After every
interrupt is
serviced the
status register
must be cleared.

0x004 INTC_CSR_ENABL [31:1] RO 00 Reserved.

E
[0] RW 00 For every
interrupt asserted
the enable should
continued...

MACsec Intel FPGA System Design User Guide Send Feedback

40
5. Configuration Registers
767516 | 2023.03.03

Offset Name Bit Type HW Reset Value Description

be asserted for
the valid
msix_data to be
sent to MCDMA.

0x08 INTC_CSR_CLEAR [31:1] RO 00 Reserved.

[0] WC 00 A clear pulse can

be generated by
writing into this
register which
clears the status.

Send Feedback MACsec Intel FPGA System Design User Guide

41
767516 | 2023.03.03

Send Feedback

6. Software Architecture
This chapter discusses software architecture.

6.1. MACsec Key Agreement Protocol

The 802.1X standard is a Port-Based Network Access Control Protocol that provides an
authentication mechanism for LAN and wireless LAN. The third edition, IEEE Std
802.1X-2010, added authenticated key agreement supporting IEEE Std 802.1AE
(MACsec).

MAC Security Key Agreement protocol (MKA -IEEE 802.1X REV-2010) is used for
discovering MACsec peers and negotiating keys.

MKA key hierarchy:

The root of the key hierarchy for any given instance of MKA is the Secure Connectivity
Association Key (CAK). For every MACsec potential peer of the same LAN, the
possession of the same CAK for the connectivity association is a must.

A CAK can be obtained in the below ways:

• It can be a pre-shared key (PSK).
• Or it can use EAP for automatic CAK management.

Each CAK is identified by a secure Connectivity Association Key Name (CKN) that
allows each of the MKA participants to select which CAK or CAK-derived key, to
process a received MKPDU.

Every key used by MKA is derived from the CAK. MKA does not use this CAK directly, it
derives two further keys, namely:
• The ICV Key (ICV): It is used to verify the integrity of MPDUs and to prove that
the transmitter of the MKPDU possesses the CAK.
• The Key Encrypting Key (KEK): It is used by Key Server which is elected by MKA,
to transport a succession of Secure Association Keys (SAKs) to the other members
of a Connectivity Association (CA).

The Key Server uses these ICK and KEK to transport/distribute the SAKs. Here, a Key
Server is elected based on the lower priority among the peers.

MKA transport with pre-shared Key:

Figure 31. MKA Transport with PSK

MKA Key Distribution

Pre-shared Key CKN Exchange and Key Server Secure Data Exchange
SAK Distribution
Configuration ICV Validation Selection with MACsec Peers

Intel Corporation. All rights reserved. Intel, the Intel logo, and other Intel marks are trademarks of Intel
Corporation or its subsidiaries. Intel warrants performance of its FPGA and semiconductor products to current
specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any ISO
products and services at any time without notice. Intel assumes no responsibility or liability arising out of the 9001:2015
application or use of any information, product, or service described herein except as expressly agreed to in Registered
writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying
on any published information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
6. Software Architecture
767516 | 2023.03.03

Pre-shared keys (CAK) are configured on MACsec enabled devices. Once peer
authentication is done, Connectivity Association is formed between the peers. Further,
the peers exchange CKN and validate ICV with the pre-shared keys.

Key sever election takes space based on the priority and it generates and distributes
SAKs. Peers then use these SAKs to encrypt the data traffic and forwards it over the
protected link.

6.2. Driver Functional Requirements

The MACsec IP driver:
• Implements the standalone MACsec IP Kernel Mode Driver for internal testing
using CLI debug tool.
• Implement the MACsec IP Kernel Mode Driver APIs.

The following functional features are supported from the MACsec IP APIs:
• Able to initialize and reset the MACsec HW IP.
— SW reset, port and SA reset.
— Initialize all the CSR registers.
• Able to set and get the Tx/Rx registers.
• Able to configure the TX/RX rekeying path.
• Able to configure the dynamic port.
• Single/multi packet mode support.
• Priority of the port grant is based on CSR setting.
• Able to set "cut through OR store/forward mode".
• Able to register the Interrupt Service Routine (i.e. interrupts are generated by
MACsec HW e.g. Packet error, Crypto errors etc.).

6.3. Software Requirements

The network stack supports:
• IEEE 802.1X - MACsec Key Agreement (MKA) packets.
• Configuring the CSR registers from the host.

6.4. Software Overview

For MACsec IP, the software stack is shown below. It depicts end-to-end flow of control
and data packets from userspace applications to the hardware layer.

Send Feedback MACsec Intel FPGA System Design User Guide

43
6. Software Architecture
767516 | 2023.03.03

Figure 32. MACsec Complete Stack with Control and Data Path

Userspace
Data Traffic wpa_supplicant
Applications IP Tool MACSec Driver Interface IEEE_802.1x (MKA)
CLI Debug Tool
(eg: ping/iperf) Request Configuration
Params L2_packet_send/recv()

Data Tx/Rx Control Commands

Socket Interface (raw_sock_netlink/rtnllink)

Kernel
Protocol Handlers (TCP/UDP/IP)

Read/Write
commands from
Linux MACSec Driver EAPol MKA CLI Debug Tool
Packets
CSR Config
MACSec IP APIs
Data Path Control Path
(Future Scope) Network Driver (McDMA) Debug Path

Hardware
MACSec FPGA

Current Scope Future Scope Open Source Intel Proprietary Patched Open Source

Note:
Future Scope – Currently not handling Data Path
Intel Proprietary – Modules which are under development stage and will be outsourced post the develoment phase.
Patched Open Source – Open source modules which will be up-streamed with few aditional features.

Control Path: Software tools like the CLI, IP route tool and wpa_suppliant are used to
trigger control commands to configure the CSRs and initiate static/dynamic key
exchanges via the socket interface.

The IP tool uses Rtnllink and Netlink APIs to create the MACsec net interface and
configure secure association details into the hardware. These calls are later handled by
the Linux MACsec Driver and trigger the macsec_ops() of the McDMA driver to
configure the CSR region.

The wpa_supplicant uses a raw packet to send or receive EAPol packets transferred
between MACsec peers for authentication/session establishment and key exchange.
These packets are directly handled by the base network driver (McDMA) to do xmit()
or recv() to/from the hardware queues. The wpa_supplicant also uses Rtnllink and
Netlink APIs to set or create the MACsec net interface and configure secure association
details into the hardware via Linux MACsec Driver and McDMA Driver(The MACsec IP
APIs are integrated with McDMA Driver).

The CLI debug tool uses Rtnllink and Netlink APIs to set or create the MACsec net
interface and configure secure association details into the hardware. These calls are
later handled by the MACsec IP Driver(with/without McDMA driver based on McDMA IP
in hardware) to configure the CSR region.

Data Path: Utilities like ping, sftp, scp, etc. are used to initiate the data transfers from
userspace to hardware. Depending upon the associated network protocol used by the
data transfer utility, respective protocol handler gets triggered based on send() and
recv() calls from application to physical layer or vice versa.

The data path is handled by the MACsec IP and SW modules are used as bypass the
data flow to above or below layers.

MACsec Intel FPGA System Design User Guide Send Feedback

44
6. Software Architecture
767516 | 2023.03.03

In case offload is off, the data path is handled by Linux MACsec Driver as Tx and Rx
path.

Tx Path: The Linux MACsec Driver adds the MACsec header with 0x88e5 protocol
number and other details to the frame. It forwards the encrypted/protected frame to
the base network driver. This network driver copies the packet to the HW Queue.

Rx Path: On the arrival of a packet, the base network driver (McDMA) allocates the
socket buffer (skb) and forwards it to the Linux MACsec driver. The MACsec Driver
takes care of decrypting the packet and delivers it to the upper layers.

The following sections discuss the major modules of software stack.

6.4.1. McDMA Driver Kernel Module

The McDMA Kernel Driver scans the devices and identify McDMA device based on the
vendor ID and device ID, enables the bus mastership and maps the BAR area of
corresponding devices by using linux existing PCI framework. As part of the probing, a
driver creates the Ethernet interface and registers the device to linux network
framework.

The McDMA Driver has the following components which include MACsec IP APIs. The
MCDMA netdev driver is used with a few additional changes made to incorporate the
MACsec IP APIs.
1. ifc_mcdma_netdev_init_module()
This is the module init function of McDMA kernel module. It is called when module
is inserted and has the following functionalities:
a. Registers generic netlink family.
b. Registers McDMA pci driver structure.
c. Allocated the memory for MACsec IP structure.
2. ifc_mcdma_netdev_exit_module()
This is module exit function of McDMA kernel module. It is called when module is
removed. It unregisters netlink family and pci driver structure.
3. ifc_mcdma_probe()
This is the pci driver hardware probe routine. It does the following:
a. Scans the devices and identify device based on vendor ID and device ID.
b. Maps the BAR regions of corresponding devices by using linux existing PCI
framework.
c. Maps available interrupts and registers MACsec interrupt-handler function.
d. Enables PCI device.
4. ifc_mcdma_remove()
This is the pci driver hardware remove routine. It does the following:
a. Unregisters the interrupts.
b. Unmaps the BAR regions.
c. Disables PCI device.
5. ifc_mcdma_netdev_open()
This function does the following:

Send Feedback MACsec Intel FPGA System Design User Guide

45
6. Software Architecture
767516 | 2023.03.03

a. Iterate through exiting interfaces to get MACsec type dev.

b. Gets linked interface/lower interface of the MACsec interface.
c. Sets Offload feature and MACsec Ops i.e intel_macsec_ops().
d. Initialize the MACsec IP i.e to default CSRs and does the soft-reset.

6.4.2. MACsec IP APIs

The MACsec IP APIs are exposed/included with McDMA(or can be added to any
network driver).

The MACsec IP APIs are available to use by McDMA for the following:
1. MACsec Ops
The APIs below are implemented to perform MACsec control path functionality.
2. intel_macsec_isr()
This is the interrupt service routine registered for MACsec ip interrupts. On
receiving an Interrupt, this signals user which interrupt has been raised. After
signaling it clears the Interrupt line and clears the interrupt status register.
3. genl_ppbb_read_reg() & genl_ppbb_write_reg()
The netlink support is added here to get the netlink request from CLI tool and
response it. In the reference design, the data path is through packet generator
and checker. The CLI debug tool is used to configure the packet generator using
netlink support.
4. Netlink functions
The netlink functions are added to McDMA driver to support register access. In the
reference design, the data flow is from the packet generator/checker. Therefore,
register settings are required from user application (i.e CLI Debug Tool). The linux
kernel module registers a generic netlink family name, on which the kernel routes
any user-space interaction. Linux kernel can have multiple netlink families
registered at a time. User-space application has to mention a corresponding name
to interact with a particular netlink channel. The MACsec IP (i.e McDMA driver)
kernel module registers the "intel_macsec" netlink family name, which is used by
CLI to interact with the kernel module.
The linux kernel module registers 6 handlers for all of the above netlink
commands. These handler functions are as follows:
a. genl_get_attr()
b. genl_set_attr()
c. genl_get_sa_attr()
d. genl_set_sa_attr()
e. genl_ppbb_read_reg()
f. genl_ppbb_write_reg()
g. genl_read_reg()
h. genl_write_reg()
All of the above handlers receive socket buffer information from the CLI and
invoke the respective MACsec IP API. On success, they return the data received
from the API. On failure, they return the error code back to CLI.

MACsec Intel FPGA System Design User Guide Send Feedback

46
6. Software Architecture
767516 | 2023.03.03

6.4.2.1. SDK

The SDK directory structure is shown below. It uses the MACsec IP driver via
management interface and provides all management functionalities to application
running on the host.

MACsec Code structure and repo details:

- drivers.platform.macsec.ipdriver

-- src

-- include

-- mcdma

-- linux_macsec

-- iptool

-- wpa_supplicant

-- utest

-- docs

-- Makefile

The new features of the MACsec IP for iptool is available in iptool.patch. This patch
can be applied to test the MACsec IP.

The new features of the MACsec IP for wpa supplicant is available in the
wpasuplicant.patch. This patch can be applied to test the MACsec IP.

Send Feedback MACsec Intel FPGA System Design User Guide

47
6. Software Architecture
767516 | 2023.03.03

6.4.3. Linux MACsec Driver Kernel Module

Figure 33. MACsec Stack Without MKA

IP Tool
Request to create/set macsec dev & configure
secure association/channel
Rtnlnetlink & NetLink Interface to process Rescue Mode
the incoming request
Linux MACSec Driver Error Handling

Data Path (Bypassed Register Interrupt

Request for macsec_ops() callback functions from IP driver) Handlers
Macsec_ops() to set/get the tx/rx attributes
set_attr() get_attr() set_sa_attr() get_sa_attr()
Enabling Macsec Offload
MACSec IP Driver + McDMA Driver capability for netdev
Macsec_ops()
Register_Write Register_Read registeration
CSR Register

SADB MACSec HW FPGA

Soft/Hard IP Ports

The Linux MACsec Driver acts as communicating medium between userspace

applications. For example, IP Tool or WPA supplicant and MACsec IP + McDMA driver.

It exposes Netlink and RtnlLink interface for interaction with the IP Tool and uses
macsec_ops() functionality to interact with the IP driver.

The Linux MACsec driver is responsible for creating the macsec0 interface over a
parent interface mentioned in the IP tool command line. It uses the created interface
to configure or get the Tx/Rx secure association details by calling respective
macsec_ops().

6.4.4. IP Tool
The IP Tool is used to configure transmit/receive secure associations and channels. It
configures the IEEE Std 802.1AE (MAC security) keys for a particular MACsec type
interface.

The Linux MACsec driver and IP Tool uses rtnetlink and genetlink APIs to handle the
input commands. Rtnetlink creates and sets up the net device whereas the genetlink
APIs are used to set up transmit and receive secure associations on a MACsec device.

The MACsec IP driver provides read/write APIs to configure the CSR region of the
MACsec HW FPGA.

MACsec Intel FPGA System Design User Guide Send Feedback

48
6. Software Architecture
767516 | 2023.03.03

6.4.5. WPA Supplicant

Figure 34. MACsec Stack with MKA

wpa_supplicant

IEEE_802.1x (MKA) MACsec Driver Interface

Request to create macsec dev
L2_packet_send() L2_packet_receive() & update tx/rx data
EAPol Packets for MKA

Socket Interface Rtnlnetlink & NetLink Interface

Control Path
Linux MACsec Driver
MACsec IP Driver

Network Driver (McDMA)

MACSec HW FPGA

The wpa_supplicant supports the MACsec Key Agreement (MKA) protocol which is
used to set up the required secure channels and associations and to perform key
exchanges between different MACsec peers.

In the above diagram, the wpa_supplicant is used for:

1. Key management and re-generation purpose.
2. Configure control path.

Initially, authentication exchanges are done using the EAPoL packets. The
wpa_supplicant constructs the MKPDU and uses a raw_packet socket interface to send
the Tx EAPol announcement to the Ethernet driver. The Ethernet driver forwards these
packets to the ring buffer and eventually over the network.

Once a MACsec peer acknowledges the received Tx announcement, it sends the Rx

EAPol packet. The Ethernet driver receives these packets. The Ethernet driver
allocates an skb and hands over the packet to the wpa_supplicant. The
wpa_supplicant decodes the packet and validates whether a potential peer sends it or
a new MKPDU frame does it.

6.5. MACsec IP APIs Sequence

This chapter covers MACsec IP APIs sequence.

6.5.1. MACsec Initialization Sequence

To bring up a port as a MACsec control port, there is an initialization sequence that
needs to be followed. All of the configuration that needs to happen as part of the
initialization sequence can be programmed using below sequence.

6.5.1.1. MACsec Reset Sequence

MACsec initialization is required to initialize the HW, reset the global, port and SA
configuration registers.

Send Feedback MACsec Intel FPGA System Design User Guide

49
6. Software Architecture
767516 | 2023.03.03

Figure 35. MACsec Initialization Flow

Start
macsec_initialize

Global Reset

Global Reset
Done?
Yes

Reset the Port

Registers

Port Reset
Done?
Yes
No
Reset the SA
Registers

Port Reset
Done?
Yes

End
macsec_initialize

6.5.1.1.1. GLOBAL Reset

Two functionalities supported in reset are listed below:

• The MACsec IP is reset through the subsystem_cold_rst_n assertion/
deassertion. There are 2 additional resets, app_ip_lite_areset_n and
app_ip_st_areset_n, which can be triggered to reset the CSR block and the
remaining logic blocks respectively. A programmable counter counts down upon
the subsystem_cold_rst_n assertion/deassertion and assert/deassert
subsystem_cold_rst_ack_n when the counter = 0.
• The MACsec IP implements a software reset in the CSR register. The software
resets the core logic, leaving the configuration registers unchanged. The MACsec
IP is in the reset state if the CSR bit is set to 0. While in this state, no traffic is
allowed in either the MACsec IP direction. The GLOBAL_SW_RESET register is used
to reset the software.

6.5.1.1.2. PORT Reset

For port reset, update the registers below:

MACsec Intel FPGA System Design User Guide Send Feedback

50
6. Software Architecture
767516 | 2023.03.03

• Set “Control port enable” to False (default value is False).

• Program the per-MACsec instance configuration:
— Set all the global stats counters to 0x0 (default is 0x0).
— Set the key length (False – 128 bits, True – 256 bits) for the MACSEC instance
associated with the port.
— Set the extended packet numbering mode for the MACsec instance associated
with the port. (False – regular packet numbering, True – extended packet
numbering).
— Zeroing port SAs (GLOBAL_ZERO CSR)
— (Optional) Set the confidentiality offset for the MACSEC instance (default is
0x0).

6.5.1.1.3. Secure Association (SA) Reset

For SA reset, update the below registers to do secure association reset.

• Set “SA_EN_TRN (Tx) or ENABLE_RECEIVE (Rx)” to False (default value is False).
• Program the per-SA instance configuration:
— Set all the SA stats counters to 0x0 (default is 0x0).
— Set the SAs (e.g. KEY, SALT, SSCI etc.) to 0.

Send Feedback MACsec Intel FPGA System Design User Guide

51
6. Software Architecture
767516 | 2023.03.03

6.5.1.2. TX Configuration Sequence

Figure 36. MACSEC set_attr() and set_sa_attr() Flows

Start Start
macsec_set_attr() macsec_set_sa_attr()

Validate port Validate port, sc, sa

attribute params attribute params

Validate Fails? Validate Fails?

No No

Validate attr value Validate attr value

range range with/without
xpn mode

Validate Fails? Validate Fails?

Get the Address Yes Get the Address Yes

Register Write Register Write

Write Fails? Write Fails?

No No

Stop Stop

MACsec Intel FPGA System Design User Guide Send Feedback

52
6. Software Architecture
767516 | 2023.03.03

Figure 37. MACSEC get_attr() and get_sa_attr() Flows

Start Start
macsec_get_attr() macsec_get_sa_attr()

Validate port Validate port, sc, sa

attribute params attribute params

Validate Fails? Validate Fails?

No No

Get the Address Get the Address

Yes Yes
Register Read Register Read

Read Fails? Read Fails?

No No

Stop Stop

Figure 38. MACsec TX Configuration Attributes

TX Path Configuration:
Below are the attributes for set_attr() or set_sa_attr()
– Set PN Limit value
– Set max packet bytes supported
– Set SCI value
– Set key value
– Set next PN value
– Set confidentiality offset
– Reset all statistic counters
– Set SA enable
– Set control port enable for the port

The set_attr() or set_sa_attr() flow diagram will be

repeated for all the attributes

Do the following:
• Set the Tx basic configuration for the MACsec instance for the port.
• Set the packet numbering limit value for the MACsec instance.
• Set the maximum packet bytes supported value for the MACsec instance.
• Set SCI value for the port.

Send Feedback MACsec Intel FPGA System Design User Guide

53
6. Software Architecture
767516 | 2023.03.03

• Choose a security association and program the following configuration:

— Set the rx Key value for the SA.
— Set the next packet number value for the SA.
— Set the confidentiality offset value for the SA.
— Initialize all the stat configurations.
• Set the “SA Enable” value to the chosen security association.
• Set “Control port enable” to True for the port.

6.5.1.3. RX Configuration Sequence

Figure 39. MACsec RX Configuration Attributes

RX Path Configuration:
Below are the attributes for set_attr() or set_sa_attr()
– Set replay window length
– Set default SCI per port
– Set SCI value for security channel
– Set key value for SA belonging to SC
– Set next PN value
– Set lowest PN value
– Reset all statistic counters
– Set SA enable
– Set control port enable for the port

The set_attr() or set_sa_attr() flow diagram will be

repeated for all the attributes

Do the following:
• Set Rx basic configuration for the MACsec instance.
• Set the replay window length if the Replay protect is enabled for the MACsec
instance.
• Set the default SCI per port.
• Program the security channel that is used on the lane by configuring:
— Set the SCI value for the security channel.
— Initialize all the stats configuration.
• Choose a security association and programming the following configuration:
— Set the Key value for the SA belonging to the SC.
— Set the next packet number value for the SA belonging to the SC.
— Set the lowest PN value for SA belonging to the SC.
— Initialize all the stats configuration.
• Set the “SA Enable” value to the chosen security association. If there are multiple
security associations programmed on Rx, enable them.
• Once all of the above is programmed, enable the control port.

MACsec Intel FPGA System Design User Guide Send Feedback

54
6. Software Architecture
767516 | 2023.03.03

6.5.1.4. TX Rekeying Sequence

When PN is about to expire, rekeying occurs and below is an example of the rekeying
sequence. All of the configuration that needs to happen as part of the rekeying
sequence can be programmed.
• Set “Enable Transmission enable” to False (default value is False) for new SA.
• Program the per-MACsec instance configuration and Tx Configuration Sequence
(section 5.1.2) for the new SA.
• Ensure no TX traffics entering MACsec IP which using expire SA.
• Set “Enable Transmission enable” to False for expire SA.
• Set “Enable Transmission enable” to True for new SA.

6.5.1.5. RX Rekeying Sequence

When PN is about to expire, rekeying occurs and below is an example of the rekeying
sequence. All of the configuration that needs to happen as part of the rekeying
sequence can be programmed.
• Set “Enable Receive enable” to False (default value is False) for new SA.
• Program the per-MACsec instance configuration and Rx Configuration (section
5.1.3) for the new SA.
• Set “Enable Receive enable” to True (default value is False) for new SA.
• Ensure RX traffics entering MACsec IP which using new SA is observed.
• Set “Enable Receive enable” to False (default value is False) for expire SA.

6.5.1.6. Cut Through/Store Forward Mode

Multi Interface Buffer supports streaming mode as well store-and-forward mode based
on CSR register settings per interface as a pseudo-static setting (no pending traffic
during mode switching). In store-and-forward mode:

Once a complete packet is received, it is available on the buffer outlet

Packet length information is calculated by the Buffer and valid on the Buffer outlet
during AXI start of packet

There is an option to discard packet if the packet contains forwarded fatal error
indicated by AXI tuser_tlast_uerr_fatal or any error in the AXI tuser_rx_client from
HSSI Sub-System.

6.5.1.7. User Single/Multi Port Settings

The USR_PORT_MULTI is to setup the single/multi port.

• ENABLE – AXI-ST Multi Packet Mode
• DISABLE – AXI-ST Single Packet Mode

6.5.1.8. Encrypt/Decrypt Port

The ENCRYPT or DECRYPT is to encrypt/decrypt the data to controlled port.

Send Feedback MACsec Intel FPGA System Design User Guide

55
6. Software Architecture
767516 | 2023.03.03

• ENCRYPT_DECRYPT
• ENCRYPT_ONLY
• DECRYPT_ONLY

6.5.1.9. Port Priority

The priority can be controlled by MACsec register as below.

• 1 — Priority assign to uncontrolled port
• 0 — Priority assign to controlled port

6.5.1.10. Interrupt Generation and Register

Based on the traffic sent to Crypto HIP, there are several errors that can be flagged
and the potential list of errors are listed below.
• Invalid AES request [0x13]
• EOB without SOB error [0x3]
• Transfer without SOB error [0x2]
• Key RAM uncorrectable error [1]
• Stream RAM uncorrectable error [0]
• AES Counter overflow indication [0x7]
• No Key received for MACsec patterns [0x17]
• No IV or tweak received for MACsec patterns [0x19]
• No end of packet for data for MACsec [0x1b]
• Crypto Core ECC error [0x20]
• FIFO Overflow [0x21]
• MAC RAM ECC error [0x22]
• Pack/Depack ECC error [0x23]

These errors are flagged through the TUSER.error_status and

TUSER.error_code signals of the AXI-ST interface. These fatal errors are not
expected since they are generated due to either invalid configuration or invalid packet
format. Any errors observed from the Crypto needs to be root-caused and fixed in
hardware or configuration.

MACsec Intel FPGA System Design User Guide Send Feedback

56
6. Software Architecture
767516 | 2023.03.03

Figure 40. MACSEC Interrupt Block

Crypto

MACsec IP
Error Next_PN
Host
Interrupt

SADB

• MACsec IP implements 2 types of interrupts:

— NextPN counters
— Error
• The SADB maintains the NextPN counter for Tx. An interrupt is generated for the
below scenarios to signal rekeying is required.
— NextPN >= Pre-exhaustion Tx PN limit CSR
— NextPN = 0 for the SA
— SA not enabled (identified through Encoding SA in TX_LANE_SC0_ENCOD_SA
CSR)
— MACsec IP stopped operation when encountering a crypto error and triggered
an interrupt.
— MACsec driver allows user call back function for interrupt handling.

6.6. Functions
This section contains functions.

6.6.1. macsec_initilize

macsec_global_reset

Synopsis static uint32_t macsec_global_reset(void)

Description Does the port reset and global sw reset.

Arguments NA

Returns MACSEC_OK if successful.

MACSEC_ERR_WRITE if CSR write failure.

Send Feedback MACsec Intel FPGA System Design User Guide

57
6. Software Architecture
767516 | 2023.03.03

macsec_port_reset

Synopsis static uint32_t macsec_port_reset(struct

macsec_info *info)

Description Does the port reset

Arguments info—points to the macsec_info structure.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_PORT if port is not valid.

MACSEC_ERR_WRITE if CSR write failure.

MACSEC_ERR_READ if CSR read failure.

macsec_sa_reset

Synopsis static uint32_t macsec_sa_reset (struct macsec_info

*info)

Description Does the secure association reset.

Arguments info—points to the macsec_info structure.

Returns MACSEC_OK if successful.

MACSEC_ERR_UNINITIALIZED if initialization failed.

MACSEC_ERR_INVALID_ARG if the MACsec_info structure or info is

NULL.

6.6.2. macsec_get_attributes

macsec_get_attributes

Synopsis int macsec_get_attr(struct macsec_info *info,

uint32_t port, enum macsec_attr attr, void *value)

Description Gets a global MACsec attribute.

Gets the MACsec attribute values that belong to the port.

Arguments info—points to the macsec_info structure.

port—MACsec instance.

attr—attribute to get the value.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_ARG if value, macsec_attr or info is NULL.

MACsec Intel FPGA System Design User Guide Send Feedback

58
6. Software Architecture
767516 | 2023.03.03

MACSEC_ERR_INVALID_PORT if the port is not initialized or is

greater than the current MACsec instances supported.

MACSEC_ERR_INVALID_ATTR if attribute is invalid.

MACSEC_ERR_READ if fails to read.

6.6.3. macsec_get_sa_attributes

macsec_get_sa_attributes

Synopsis int macsec_get_sa_attr(struct macsec_info *info,

uint32_t port, uint32_t sc, uint32_t sa, enum
macsec_sa_attr attr, void *value)

Description Retrieves a MACsec SA attribute.

Retrieves attribute for a given (sc,sa) pair for the specified port.

Arguments info—points to the macsec_info structure.

port—logical port value.

sc—secure channel value.

sa—secure association value.

attr—MACsec SA attribute.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_ARG if value, macsec_attr or info is NULL

or SC/SA invalid.

MACSEC_ERR_INVALID_PORT if port not associated with any port.

MACSEC_ERR_INVALID_ATTR if the attribute is unknown or the key

or salt attribute is tried to be read.

MACSEC_ERR_READ if fails to read the csr register.

6.6.4. macsec_set_attributes

macsec_set_attributes

Synopsis int macsec_set_attr (struct macsec_info *info,

uint32_t port, enum macsec_attr attr, const void
*value)

Description Sets a global MACsec attribute.

Gets the MACsec attribute values for belonging to the port.

Send Feedback MACsec Intel FPGA System Design User Guide

59
6. Software Architecture
767516 | 2023.03.03

Arguments info—points to the macsec_info structure.

port—MACsec instance.

attr—MACsec attribute to set the value.

value—points to the value to set for the attribute.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_ARG if value, macsec_attr or info is NULL.

MACSEC_ERR_INVALID_PORT if the port is not initialized or is

greater than the current MACsec instances supported.

MACSEC_ERR_INVALID_ATTR if attribute is invalid.

MACSEC_ERR_INVALID_VALUE if value is out of range.

MACSEC_ERR_WRITE if fails to write.

6.6.5. macsec_set_sa_attributes

macsec_set_sa_attributes

Synopsis int macsec_set_sa_attr (struct macsec_info *info,

uint32_t port, uint32_t sc, uint32_t sa, enum
macsec_sa_attr attr, const void *value)

Description Sets a MACsec SA attribute.

Sets attribute for a given (sc,sa) pair for the specified port.

Arguments info—points to the macsec_info structure.

port—logical port value.

sc—secure channel value.

sa—secure association value.

attr—MACsec SA attribute.

value—points to the value to set for the attribute.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_ARG if value, macsec_attr or info is NULL

or SC/SA invalid.
MACSEC_ERR_INVALID_PORT if port not associated with any port.

MACSEC_ERR_INVALID_ATTR if the attribute is unknown or the key

or salt attribute is tried to be read.

MACSEC_ERR_INVALID_VALUE if the MACsec instance is invalid.

MACsec Intel FPGA System Design User Guide Send Feedback

60
6. Software Architecture
767516 | 2023.03.03

MACSEC_ERR_WRITE if fails to write to the csr register.

6.6.6. macsec_read_register

macsec_read_register

Synopsis uint32_t macsec_read(struct macsec_info *info,

uint32_t offset, uint32_t port, uint32_t sc,
uint32_t sa, void *value)

Description Read the MACsec IP register.

Arguments info—points to the macsec_info structure.

port—physical port.

offset—CSR register offset value.

sc—secure channel value.

sa—secure association value.

value—read value.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_PORT if port not associated with any port.

MACSEC_ERR_INVALID_ARG if the macsec_info structure or info is

NULL or offset and SC/SA are invalid.

MACSEC_ERR_READ if CSR read failure.

6.6.7. macsec_write_register

macsec_write_register

Synopsis uint32_t macsec_write(struct macsec_info *info,

uint32_t offset, uint32_t port, uint32_t sc,
uint32_t sa, uint64_t value)

Description Write the value to CSR register.

Arguments info—points to the macsec_info structure.

port—port number.

offset—CSR register offset value.

sc—secure channel value.

sa—secure association value.

Send Feedback MACsec Intel FPGA System Design User Guide

61
6. Software Architecture
767516 | 2023.03.03

value—write value.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_PORT if port not associated with any port.

MACSEC_ERR_INVALID_ARG if the macsec_info structure or info is

NULL or offset and SA/SC are invalid.

MACSEC_ERR_WRITE if CSR write failure.

6.6.8. macsec_set_port_configuration

macsec_set_port_configuration

Synopsis int macsec_set_port_config(int port, int set_mode)

Description Sets the port to Cut Through mode OR Store and Forward mode.

Port with traffics that required integrity protection only packet

needs to be configured as Store and Forward.

Arguments port—physical port.

set_mode—sets the port to Cut Through mode OR Store and

Forward mode.
• 0—Cut Through mode
• 1—Store and Forward mode

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_PORT if port is not valid.

MACSEC_ERR_INVALID_MODE if mode is not valid.

6.6.9. macsec_rate_configuration

macsec_rate_configuration

Synopsis iint macsec_rate_config(int port, int data_rate)

Description Ethernet Dynamic data rate changes support through Port Mux/
Demux

Arguments port—physical port.

data_rate—data rate for Ethernet port.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_PORT if port is not valid.

MACsec Intel FPGA System Design User Guide Send Feedback

62
6. Software Architecture
767516 | 2023.03.03

MACSEC_ERR_INVALID_DATA_RATE if data rate is not valid.

6.6.10. macsec_single_or_multi_port

macsec_single_or_multi_port

Synopsis int macsec_single_or_multi_port(int port, int

multi_port)

Description Sets the single or multiple port support.

Arguments port—physical port.

multi_port—if multi_port is true, sets to multiple port; if multi_port

is false, sets to single port.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_PORT if port is not valid.

MACSEC_ERR_INVALID_MULTI if multi port is not supported.

6.6.11. macsec_crypto_mode

macsec_crypto_mode

Synopsis int macsec_crypto_mode(int port, int crypto_mode)

Description Sets the encryption and decryption to transmit and receive port
resp.

Arguments port—physical port.

crypto_mode—
• if crypto_mode is ENCRYPT_DECRYPT, set both TX/RX ports.
• if crypto_mode is ENCRYPT_ONLY, set to only TX port.
• if crypto_mode is DECRYPT_ONLY, set to only RX port.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_PORT if port is not valid.

MACSEC_ERR_INVALID_CRYPTO_MODE if crypto_mode is not

invalid.

Send Feedback MACsec Intel FPGA System Design User Guide

63
6. Software Architecture
767516 | 2023.03.03

6.6.12. macsec_port_priority

macsec_port_priority

Synopsis int macsec_port_priority(int port, int

priority_mode)

Description Sets the priority of controlled and uncontrolled ports.

Arguments port—physical port.

priority_mode—
• if priority_mode is 0, sets both equal priority.
• if priority_mode is 1, sets controlled port has high priority.
• if priority_mode is 2, sets uncontrolled port has high priority.

Returns MACSEC_OK if successful.

MACSEC_ERR_INVALID_PORT if port is not valid.

MACSEC_ERR_INVALID_PRIORITY if priority is not invalid/

supported.

6.6.13. macsec_register_isr

macsec_register_isr

Synopsis int macsec_register_isr

Description Registers the ISR function.

Arguments NA

Returns MACSEC_OK if successful.

MACSEC_ERR_UNSUPPORTED if the function not registered.

6.7. Software Tools

This section covers software tools which include the IP tool and WPA_supplicant.

MACsec Intel FPGA System Design User Guide Send Feedback

64
6. Software Architecture
767516 | 2023.03.03

6.7.1. IP Tool
Figure 41. Configuration and Software Stack for IP Tool

IP Tool commands to
IP Tool
send port, sc, sa, key etc

Abstract Layer MACSec Functionality

Linux MACSec Driver E.g. Initialization, Tx/Rx configuration
flow etc

MACSec IP driver to provide the APIs access

MACSec IP Driver + McDMA Driver to above layer and write/read to HW

HW FPGA (MACSec IP) MACSec Hardware IP

IP tool: It can be used to statically configure the Tx/RX information and keys of the
interface.

Linux MACsec: This driver triggers one of the macsec_ops() API depending on the
input received via netlink interface.

MACsec IP Driver: It exposes macsec_ops() to linux MACsec driver and performs

register read writes for the input provided.

HW FPGA: MACsec IP in FPGA and HW Crypto.

The examples below are commands used for configuration and testing:

Create a MACsec device on link eth0 (offload is disabled by default):

# sudo ip link add link eno0 MACsec0 type MACsec encrypt on

Configure a secure association on that device:

# ip MACsec add MACsec0 tx sa 0 pn 1024 on key 01

81818181818181818181818181818181

Configure a receive channel:

# ip MACsec add MACsec0 rx port 1234 address c6:19:52:8f:e6:a0

Configure a receive association:

# ip MACsec add MACsec0 rx port 1234 address c6:19:52:8f:e6:a0 sa 0 pn 1 on key

00 82828282828282828282828282828282

Display MACsec configuration:

# ip MACsec show

Send Feedback MACsec Intel FPGA System Design User Guide

65
6. Software Architecture
767516 | 2023.03.03

Configure offloading on an interface:

# ip MACsec offload MACsec0 mac

Configure offloading upon MACsec device creation:

# ip link add link eno0 MACsec0 type MACsec port 11 encrypt on offload mac

6.7.2. WPA_Supplicant
Figure 42. WPA_Supplicant on Two Hosts

Host 1 Host 2
wpa_supplicant wpa_supplicant
Control Path MKA Key MKA Key
Exchange Control Path
Exchange
Kernel Kernel

Step 1: MKA mutual authentication

Step 2: Secure Channel Establishment
Step 3: Key Generation
Step 4: Key & Secure association setup

Wpa_supplicant: It uses a config file that includes pre-shared CAK and CKN keys on
both hosts. Two peers achieve mutual authentication via exchanging MKA keys. The
MACsec Key Agreement protocol uses EAPoL PDUs to transmit and receive MKPDUs
securely among each other.

Secure associations using these keys are configured on both hosts. The
wpa_supplicant translates the information derived through MKA and configures the
kernel's MACsec implementation.

Kernel: It configures the CSR region, and when traffic is initiated, it sends packets
protected by MACsec on the "MACsec0" interface, which is a separate network device
dedicated to encrypted traffic.

Steps 3 and 4 (as mentioned in the above diagram) are later repeated (as many times
as necessary) while wpa_supplicant keeps running to transition to a new key when the
current key expires.

Wpa_supplicant uses the configuration file below:

ctrl_interfaces=/var/run/wpa_supplicant

eapol_version=3
ap_scan=0
fast_reauth=1

# Example Configuration for MACsec with preshared key

network={
key_mgmt=NONE
eapol_flags=0
macsec_policy=1

MACsec Intel FPGA System Design User Guide Send Feedback

66
6. Software Architecture
767516 | 2023.03.03

mka_cak=0123456789ABCDEF123456789ABCDEF
mka_ckn=6162636465666768696A6B6C6D6E6F707172737475767778797A303132333435
mka_priority=2
macsec_integ_only=0
macsec_port=0
macsec_replay_protect=1
macsec_replay_window=50
#Newly_added
macsec_val_frames=2
mka_cipher_suit="GCM-AES-XPN-256"
macsec_ssci=0xABCD
macsec_scb=0
macsec_es=0
macsec_send_sci=1
}

Command: ./wpa_supplicant -i ens801f0 -D macsec_linux -c

wpa_supplicant_MACsec.conf

Where, -i: Interface to be used; -D: Driver to be used; -c: Config file.

6.7.3. CLI Debug Tool

The CLI Debug Tools is an application for debugging purposes and configures the
individual registers. This application does the sharing of keys and the configuration of
MACsec IP registers. This information is then passed to the MACsec IP Driver, which
does the MACsec HW configuration via McDMA Driver.

The MACsec IP driver and McDMA provides the MACsec control path APIs to read/write
the SADB registers. Based on these register configuration of the MACsec data path is
performed.

6.7.3.1. Functional Requirements

Intel provides a command line software debug tool for debugging and testing
purposes. The tool can be used at the Host to support the features below.
• Implemented to test the functions of the MACsec IP driver.
• Implemented to manage with CLI (command line interface) the system example
design and its tests.

6.7.3.2. Features Overview

The debug tool is a command line interface application that provides below-mentioned
capabilities to the Host/HPS.
• Debug tool help
• Debug tool version
• Initialize the MACsec IP
• Software reset
• Get the global and port attributes
• Set the global and port attributes
• Get the per SA attributes
• Set the per SA attributes
• Read any register value

Send Feedback MACsec Intel FPGA System Design User Guide

67
6. Software Architecture
767516 | 2023.03.03

• Write value to any register

• Tx/Rx Configuration
• Tx/Rx Rekeying Configuration
• Dynamic Reconfiguration of ports
• Cut Through or Store/Forward
• Dump all statistic registers

6.7.3.3. Design with HOST

The figure below shows the CLI debug application diagram which can be used for
debugging, key generation, and configuring the MACsec IP. This information is passed
onto the MACsec IP Driver and then to McDMA driver to do the MACsec HW
configuration.

The HW FPGA has the MACsec IP and Soft/Hard Crypto logic. This HW does the
encryption/decryption and insertion/deletion of MACsec Header to packet data as per
the configuration from MACSEC IP driver. The MACsec can be enabled/disabled from
driver. If enabled, it uses the controlled port to transmit/receive the data otherwise
uses the uncontrolled port.

Figure 43. CLI Implementation Stack

CLI Debug Tool

Request for set/get the commands

NetLink Interface to Receive/Transmit data

set_attr() get_attr() set_sa_attr() get_sa_attr()

MACSec IP Driver + McDMA Driver

register_write register_read

CSR Register
MACSec HW FPGA
SADB

Soft/Hard IP Ports

In MACsec IP driver is a kernel mode driver and uses a netlink interface to

communicate with above layer. The driver exposes 4 APIs to set/get the port
attributes OR set/get the SC/SA attributes. In the MACsec IP driver, the management/
AXI-Lite interface is used to read/write the CSR registers.

In the FPGA, encryption/decryption are controlled from the SADB registers which are
updated from the MACsec IP driver.

MACsec Intel FPGA System Design User Guide Send Feedback

68
6. Software Architecture
767516 | 2023.03.03

6.7.3.4. Netlink Interface

The MACsec IP driver accesses memory mapped MACsec IP registers over the AXI bus.
This module communicates with user-space over netlink socket. The details of netlink
structure, family, and API's are as follows:

NETLINK STRUCTURE

The CLI talks with linux kernel module over a netlink communication channel. The
netlink family used to establish this communication is "GENERIC_NETLINK". There are
4 main netlink commands used for configuring the MACsec IP. These are:
1. INTEL_MACSEC_C_GETATTR
2. INTEL_MACSEC_C_SETATTR
3. INTEL_MACSEC_C_GETSAATTR
4. INTEL_MACSEC_C_SETSAATTR

The above netlink commands are responsible for handling macsec_get_attr(),

macsec_set_attr(), macsec_get_sa_attr() and macsec_set_sa_attr()
APIs based on invokation from the CLI.

The 2 additional commands, which are used to write/read the MACsec PPBB IP device
registers directly, are:
1. INTEL_MACSEC_C_PPBB_READREG
2. INTEL_MACSEC_C_PPBB_WRITEREG

Apart from the commands above, there are 2 additional commands, which are used to
peek-and-poke MACsec IP device registers directly. This read-write functionality is
helpful in debugging. These commands are:
1. INTEL_MACSEC_C_READREG
2. INTEL_MACSEC_C_WRITEREG

The netlink protocol is a socket based communication. The socket attributes used for
exchange of information between the CLI and Kernel are:
1. INTEL_MACSEC_A_PORT
2. INTEL_MACSEC_A_ATTR
3. INTEL_MACSEC_A_RW_VAL
4. INTEL_MACSEC_A_SC
5. INTEL_MACSEC_A_SA

The above netlink attributes correspond to the MACsec ip port value, MACsec ip
command attribute, read/write value for a particular command attribute, MACsec ip
secure channel value, and MACsec ip secure association value.

Apart from the main attributes above there is 1 additional attribute used for
debugging. This helps program offset the device register to be programmed. The
attribute is:
• INTEL_MACSEC_A_OFFSET

NETLINK FAMILY

Send Feedback MACsec Intel FPGA System Design User Guide

69
6. Software Architecture
767516 | 2023.03.03

The linux kernel module registers a generic netlink family name, on which the kernel
routes any user-space interaction. A Linux kernel can have multiple netlink families
registered at a time. A User-space application has to mention the corresponding name
to interact with a particular netlink channel.

The MACsec IP kernel module registers with "intel_MACsec" as a netlink family name,
which is used by the CLI to interact with the Kernel module.

NETLINK HANDLER FUNCTIONS

The linux kernel module registers 6 handlers for all of the above netlink commands.
These handler functions are:
1. genl_get_attr()
2. genl_set_attr()
3. genl_get_sa_attr()
4. genl_set_sa_attr()
5. genl_ppbb_read_reg()
6. genl_ppbb_write_reg()
7. genl_read_reg()
8. genl_write_reg()

All of the above handlers receive socket buffer information from the CLI, invoke the
respective MACsec IP API, on success return the data received from the API, on failure
returns the error code back to the CLI.

6.7.3.5. SDK

The SDK directory structure for CLI Debug Tool is shown below. It provides a
reference example application to explain the API usages.

Repo details:

- drivers.platform.macsec.cli

-- src

-- include

-- docs

-- Makefile

6.7.3.6. Design Overview

The diagram below shows the configuration and design overview of the FPGA debug
tools with respect to the HPS and Host.

MACsec Intel FPGA System Design User Guide Send Feedback

70
6. Software Architecture
767516 | 2023.03.03

Figure 44. Configuration and Software Stack for Debug tool

MACSec Debug CLI + MACSec Debug CLI +
Abstract layer MACSec functionality, Abstract layer MACSec functionality,
CLI Debug Tool CLI Debug Tool
e.g. Initialization/Tx/Rx Configuration e.g. Initialization/Tx/Rx Configuration
Flow etc. Flow etc.

MACSec IP Driver + McDMA FM7/8 MACSec IP Driver MACSec IP Driver FM7/8 MACSec IP Driver

FM7/8 MACSec IP + Hard/Soft FM7/8 MACSec IP + Hard/Soft

HW FPGA (MACSec IP) HW FPGA (MACSec IP)
Crypto Crypto

Design Overview with HOST Design Overview with HPS

Debug CLI : This contains debug applications as mentioned above in the feature
overview.

MACsec IP driver: In case of the host example design, the read write calls are taken
care by the McDMA driver and for the HPS design. The MACsec IP Driver is directly
written into the HW.

Hardware FPGA: The MACsec IP in the FPGA and HW Crypto.

Below are the commands used for testing:

Cmd: ./cli_macsec sa-sc-tx-stat-encr-pckt -p 1 -c 1 -a 3 -w 0x01

Where,

p = port, c = secure channel, a = secure association, o = offset, w = write

Output:

cmd mapped to macsec_get_set_sa_attribute

Communicating over netlink socket

CMD mapped with attr: 40

Bytes written: 64

Device says: port 1 sc 1 sa 3 cmd 40 val 0x1

SUCCESS for attr: sa-sc-tx-stat-encr-pckt port: 1 sc: 1 sa: 3

The command sa-sc-tx-stat-encr-pckt is defined in command_list_file.txt

with the following rule.

sa-sc-tx-stat-encr-pckt ATTR_T_UINT32 MACSEC_SC_SA_TX_STAT_ENCR_PCKT 40

You can add new commands by appending to the command_list_file.txt. The

same command has to be checked in the cli_MACsec infrastructure. Similarly, a script
can be used to run multiple commands in order. For example,

sh test_cli_macsec.sh -p=1 -c=1 -a=3 -o=0xff2f -w=0x01

Send Feedback MACsec Intel FPGA System Design User Guide

71
767516 | 2023.03.03

Send Feedback

7. Generating the System Design

This chapter discusses how to obtain, simulate, build, and run the System Design on
hardware. The design supports different flavors of Tile combinations with different
data rate support. The design is flexible enough to be configured for an E-Tile or an F-
Tile based MAC with 25G or 100G data rate support using compile time directives as
shown in the sections below.

7.1. Software Requirements

• Intel Quartus Prime Pro Edition software version 22.4
• Supported simulator: Synopsys VCS

7.2. Obtaining the Reference Design

To obtain the Reference Design, contact Intel Support and quote case number
18027237579.

7.3. Reference Design Directory Structure

After obtaining the reference design, you must first uncompress it. The uncompressed
directory structure is shown below.
<Reference Design Name>
<HW_Release>
env/
sim/
src/
syn/
<SW_Release>
include/
src/
mcdma/
linux_macsec/
iptool/
wpa_supplicant/
utest/
patches/

Each directory in <HW_Release> is described below:

Intel Corporation. All rights reserved. Intel, the Intel logo, and other Intel marks are trademarks of Intel
Corporation or its subsidiaries. Intel warrants performance of its FPGA and semiconductor products to current
specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any ISO
products and services at any time without notice. Intel assumes no responsibility or liability arising out of the 9001:2015
application or use of any information, product, or service described herein except as expressly agreed to in Registered
writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying
on any published information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.
7. Generating the System Design
767516 | 2023.03.03

• env Directory — Contains the environment setting file which needs to be sourced
to get the latest Quartus build and the appropriate Arc Shell.
• sim Directory — Contains the subdirectories and files for simulation. The testcase
directory includes all the testcases for simulation.
runDir subdirectory — Where you run the simulation and the log files, dump
files etc, which are created inside separate subdirectories.
The tbTop and common subdirectories contain defines and other function call
files.
• src Directory — Contains all the RTL files for the design, the IP source files, and
the qsys generated files.
• syn Directory — Contains the required files for Synthesis. The major files are the
Quartus project files, a Quartus settings file, a pin constraints file, clock
constraints, and other synthesis settings files.

Each directory in <SW_Release> is described below:

• include Directory — Contains header files associated with various MACsec IP

API/driver modules.
• src Directory — Contains MACsec APIs/driver source files. They are categorized
into the functionalities below:
— MACsec IP initialization.
— MACsec IP get/set port attributes.
— MACsec IP get/set port, sc and sa attributes.
— MACsec PPBB IP component write/read CSR registers.
• mcdma Directory — Contains the MCDMA driver as network/ethernet driver.
• linux_macsec Directory — Contains the open source MACsec driver used
between the user applications (i.e. wpa_supplicant and iptool) and drivers (i.e.
mcdma driver that supports the MACsec APIs).
• iptool Directory — Contains the open source which is ip-macsec reference
implementation of application to exercise the drivers. The patch provides the new
features supported in the MACsec IP HW.
• wpa_supplicant Directory — Contains the open source which is wpa-macsec
reference implementation of application to exercise the drivers. The patch
provides the new features supported in the MACsec IP HW.
• utest Directory — Contains the src code of the unit test.
• patches Directory — Contains scripts and patches used for the HPS based design
and yocto build.

7.4. Simulation Command Arguments

The following arguments are used when running simulations:

-h — Without any other arguments gives help option for the script with an example.

Example: sh run_sim.sh -h

Send Feedback MACsec Intel FPGA System Design User Guide

73
7. Generating the System Design
767516 | 2023.03.03

• -g <value>
— 0: run simulation without generating files again.
— 1: If simulation is run for the first time or there is any change in IP file list.
• -c <value>
— 1: To clean up all temporary generated files.
• -b <value>
— 0: to run with lightweight MCDMA BFM.
— 1: to run with rootport bfm.
Note: An E-tile design supports the RootPort_BFM but an F-tile design does
not support it in this release. The RootPort_BFM enabled simulations
take longer simulation times and larger VPD dumps.
• -m <value>
— 0: run without vcd enabled.
— 1: run with vcd enabled.
• - d <"+define+macro"> (Specify the single/multiple verilog defines to be
passed)

7.4.1. Simulation -d Options

The following -d options can be used when running your simulations:

Table 11. Simulation -d Options

-d "<option>" Description

+define+MAC_SRD_CFG_25G Selects 25G Ethernet. If this macro is not specified, then

the default selection is 100G.
Note: There is no separate macro define for 100G.

+define+QSFP_EXTERNAL_LOOPBACK Selects Ethernet loopback at QSFP level.

Note: If this macro define is not specified, then the
MACsec level loopback is enabled.

+define+PCIE_USR_DATA_WIDTH_X16 Selects 16 lane PCIe.

Note: If this macro define is not specified, then it is an 8-
lane selection by default. For example: in a MACsec
SysED, an E-tile-based design is 16 lane and an F-
tile-based design is 8 lane.

+define+ENABLE_ETILE_ETH Selecting E-tile design only.

+define+ENABLE_FTILE_ETH Selecting F-tile design only.

Here are some examples of the -d usage:

1. 25G with QSFP external loopback.

-d "+define+MAC_SRD_CFG_25G +define+QSFP_EXTERNAL_LOOPBACK"
2. 25G with MACsec level loopback.
-d "+define+MAC_SRD_CFG_25G"
3. 100G with QSFP external loopback.

MACsec Intel FPGA System Design User Guide Send Feedback

74
7. Generating the System Design
767516 | 2023.03.03

-d "+define+QSFP_EXTERNAL_LOOPBACK"
4. 100G with MACsec level loopback: No need to give -d option.

7.5. Simulation Test Cases

The following test cases are located in the <project directory>/sim/Testcase
directory.

Table 12. Simulation Test Cases

Test Case Name Test Case Description

The list of testcases below contains the basic tests which include CSR and traffic test (full duplex) on both controlled and
uncontrolled ports of the MACsec IP.

basicCSRTest.sv Basic CSR Test: Test runs CSR writes and reads on the
MACsec IP and Packet Client module. This is the basic test
to be run to ensure that the CSR registers are accessible
from the host (here the testbench mimics the host machine
with two VM instances). You can modify this test. CSR
registers can be added or removed by referring to this
testcase.
Note: For any new addition you need to make sure the
CSR is declared and mapped to the correct register
offsets in /sim/tbTop/common_defines.sv.

basicDynTrafficTest.sv Basic Dynamic TrafficTest: Enables traffic flow from

packet_client_0 -> macsec_ip_0 -> ethernet_p0
-> loopback at qsfp end -> ethernet_p1 ->
macsec_ip_1 -> packet_client_1 and from other
direction. This works in a full duplex mode.
This test only sends a fixed pattern with minimum packet
size.

eapolTrafficTest.sv EAPol traffic test: Enables MKA traffic from the host. The
EAPol traffic flow is host_vm0 -> mcdma ->
macsec_ip_0 -> ethernet_p0 -> loopback at qsfp
end -> ethernet_p1 -> macsec_ip_1 -> mcdma ->
host_vm1.
In this test vm0 initiates MKA packet and vm1 validates the
reception of MKA packet within the PCIe RP_BFM.

fullDynTrafficTest.sv Full Dynamic TrafficTest: Enables traffic flow from

packet_client_0 -> macsec_ip_0 -> ethernet_p0
-> loopback at qsfp end -> ethernet_p1 ->
macsec_ip_1 -> packet_client_1 and from other
macsec direction. This works in a full duplex mode.
This test sends various min and max size packets with fixed
and incremental data patterns. You can modify this test for
different packet sizes and traffic patterns.

combinedTrafficTest.sv Combined Traffic test: Combines both basic dynamic

traffic and EAPoL traffic, i.e. both controlled and
uncontrolled ports are driven with respective traffic
simultaneously. This mimics the real traffic test scenario as
it is in a SW+HW test on board.

pcieUserMsixTest.sv PCIe user MSIX test: Checks the interrupt function. Each
MACsec IP has its own corresponding interrupt controller.
There are 3 registers (i.e. enable, clear, and status) in each
controller to manage the interrupt service. In this test case
an interrupt is forced and the test scenario is created and
tested.
continued...

Send Feedback MACsec Intel FPGA System Design User Guide

75
7. Generating the System Design
767516 | 2023.03.03

Test Case Name Test Case Description

The list of testcases below recreates the SW+HW validation tests in the RTL simulation in which they mainly check the
datapath.

rx_sc1_not_receiving_pckts.sv RX SC1 not receiving packet: Is an IP scenario test

where you configure the Tx and Rx with SC1 and SA0, and
the RX inuse register is expected to be 0x1.

fullDynTrafficTest_enc_off_pro_on_fail.sv Full Dynamic TrafficTest with Encryption ON and

Protection ON: With SCI not included is an IP scenario test
where the MACsec IP is configured with Enc_on, Pro_on,
and SCIs disabled and it is expected that the traffic has to
pass through without any issue.
Refer to this test case to see the list of MACsec IP registers
that needs to be updated.
Traffic flow: packet_client_0 -> macsec_ip_0 ->
ethernet_p0 -> loopback at qsfp end ->
ethernet_p1 -> macsec_ip_1 -> packet_client_1
and vice versa.
This test sends a basic fixed data pattern traffic.
You can modify this test to support different packet sizes
and traffic patterns.

fullDynTrafficTest_enc_on_pro_on.sv Full Dynamic TrafficTest with Encryption ON and

Protection ON: Is an IP scenario test where the MACsec IP
is configured with Enc_on and Pro_on and it is expected
that the traffic has to pass through without any issue.
Refer to this test case to see the list of MACsec IP registers
that needs to be updated.
Traffic flow: packet_client_0 -> macsec_ip_0 ->
ethernet_p0 -> loopback at qsfp end ->
ethernet_p1 -> macsec_ip_1 -> packet_client_1
and vice versa.
This test sends a basic fixed data pattern traffic. You can
modify this test to support different packet sizes and traffic
patterns.

Eyn_On_pro_on_outpkt_toolong.sv Encryption on, protection on and outpacket too long

Test: Is a scenario test where the MACsec IP is configured
with Enc_on, Pro_on and it is expected that the RX traffic
should drop if outpacket too long error is reported.
Refer to this test case to check the list of MACsec IP
registers that needs to be updated.
Traffic flow: packet_client_0 -> macsec_ip_0 ->
ethernet_p0 -> loopback at qsfp end ->
ethernet_p1 -> macsec_ip_1 -> packet_client_1
and vice versa.
This test sends a basic fixed data pattern traffic.

Eyn_off_Pro_Off.sv Encryption off and protection off Test: Is a scenario test

where the MACsec IP is configured with Enc_off, Pro_off and
it is expected that the traffic has to pass through without
any issue.
Refer to this test case to check the list of MACsec IP
registers that needs to be updated.
Traffic flow: packet_client_0 -> macsec_ip_0 ->
ethernet_p0 -> loopback at qsfp end ->
ethernet_p1 -> macsec_ip_1 -> packet_client_1
and vice versa.
This test sends a basic fixed data pattern traffic.
You can modify this test to support different packet sizes
and traffic patterns.
continued...

MACsec Intel FPGA System Design User Guide Send Feedback

76
7. Generating the System Design
767516 | 2023.03.03

Test Case Name Test Case Description

Eyn_off_pro_on_with_end_station_on.sv TrafficTest with Encryption OFF and Protection ON

with end station ON: Is an IP scenario test where the
MACsec IP is configured with Enc_off and Pro_on and it is
expected that the traffic has to pass through without any
issue.
Refer to this test case to see the list of MACsec IP registers
that needs to be updated.
Traffic flow: packet_client_0 -> macsec_ip_0 ->
ethernet_p0 -> loopback at qsfp end ->
ethernet_p1 -> macsec_ip_1 -> packet_client_1
and vice versa.
This test sends a basic fixed data pattern traffic.
You can modify this test to support different packet sizes
and traffic patterns.

Eyn_on_pro_on_with_RX_config_only_SCI.sv Encryption on, protection on with RX config only SCI

Test: Is a scenario test where the MACsec IP is configured
with Enc_on, Pro_on and RX config only SCI and it is
expected that the traffic has to pass through without any
issue.
Refer to this test case to check the list of MACsec IP
registers that needs to be updated.
Traffic flow: packet_client_0 -> macsec_ip_0 ->
ethernet_p0 -> loopback at qsfp end ->
ethernet_p1 -> macsec_ip_1 -> packet_client_1
and vice versa.
This test sends a basic fixed data pattern traffic.
You can modify this test to support different packet sizes
and traffic patterns.
continued...

Send Feedback MACsec Intel FPGA System Design User Guide

77
7. Generating the System Design
767516 | 2023.03.03

Test Case Name Test Case Description

Eyn_on_pro_on_with_cport_off.sv Encryption on, protection on with cport off Test: Is a

scenario test where the MACsec IP is configured with
Enc_on, Pro_on with cport off and it is expected that the
traffic has to pass through without any issue.
Refer to this test case to check the list of MACsec IP
registers that needs to be updated.
Traffic flow: packet_client_0 -> macsec_ip_0 ->
ethernet_p0 -> loopback at qsfp end ->
ethernet_p1 -> macsec_ip_1 -> packet_client_1
and vice versa.
This test sends a basic fixed data pattern traffic.
You can modify this test to support different packet sizes
and traffic patterns.

FullDynTest_EynON_Pro_Off.sv TrafficTest with Encryption ON and Protection OFF: Is

an IP scenario test where the MACsec IP is configured with
Enc_on, Pro_off and it is expected that the traffic has to
pass through without any issue.
Refer to this test case to check the list of MACsec IP
registers that needs to be updated.
Traffic flow: packet_client_0 -> macsec_ip_0 ->
ethernet_p0 -> loopback at qsfp end ->
ethernet_p1 -> macsec_ip_1 -> packet_client_1
and vice versa.
This test sends a basic fixed data pattern traffic.
You can modify this test to support different packet sizes
and traffic patterns.

FullDynTest_Confidentiality_offset_50bytes.sv TrafficTest with Confidentiality offset selected 0x32 :

Is an IP scenario test where the MACsec IP is configured
with cipher confidentiality offset to 50 bytes. It is expected
that the traffic has to pass through without any issue.
Refer to this test case to see the list of MACsec IP registers
that needs to be updated.
Traffic flow: packet_client_0 -> macsec_ip_0 ->
ethernet_p0 -> loopback at qsfp end ->
ethernet_p1 -> macsec_ip_1 -> packet_client_1
and vice versa.
This test sends a basic fixed data pattern traffic.
You can modify this test to support different packet sizes
and traffic patterns.

7.6. Complete Simulation Command

The following commands provide the entire syntax in order to run a simulation:

sh run_sim.sh -t <test_case_name> -g <0/1> -b <0/1> -m <0/1> -d

"+define+macro1 +define+macro2 +define+macro3"

Examples of this syntax for specific tests are shown below:

1. For E-TILE design simulation (PCIe Gen4x16)

MACsec Intel FPGA System Design User Guide Send Feedback

78
7. Generating the System Design
767516 | 2023.03.03

• 25G E-tile design, QSFP loopback, without VPD dump enable

sh run_sim.sh -t basicCSRTest -g 1 -c 1 -b 0 -m 0 -d
"+define+PCIE_USR_DATA_WIDTH_X16 +define+ENABLE_ETILE_ETH
+define+MAC_SRD_CFG_25G +define+QSFP_EXTERNAL_LOOPBACK"
• 25G E-tile design, QSFP loopback, with RP_BFM
sh run_sim.sh -t basicCSRTest -g 1 -c 1 -b 1 -m 0 -d
"+define+PCIE_USR_DATA_WIDTH_X16 +define+ENABLE_ETILE_ETH
+define+MAC_SRD_CFG_25G +define+QSFP_EXTERNAL_LOOPBACK"
• 100G E-tile, MACSEC level Loopback, with VPD dump enable
sh run_sim.sh -t basicCSRTest -g 1 -b 0 -m 1 -d "+define
+PCIE_USR_DATA_WIDTH_X16 +define+ENABLE_ETILE_ETH"
2. For F-TILE design simulation (PCIe Gen4x8)
• 25G F-tile design, QSFP loopback, without VPD dump enable
sh run_sim.sh -t basicCSRTest -g 1 -c 1 -b 0 -m 0 -d
"+define+ENABLE_FTILE_ETH +define+MAC_SRD_CFG_25G +define
+QSFP_EXTERNAL_LOOPBACK"
• 100G F-tile design, QSFP loopback, with VPD dump enable
sh run_sim.sh -t basicCSRTest -g 1 -c 1 -b 0 -m 1 -d
"+define+ENABLE_FTILE_ETH +define+QSFP_EXTERNAL_LOOPBACK"
• 100G F-tile design, MACSEC level loopback, with VPD dump enable
sh run_sim.sh -t basicCSRTest -g 1 -c 1 -b 0 -m 1 -d
"+define+ENABLE_FTILE_ETH"

7.7. Simulation Requirements

The simulation models for the Symmetric Cryptographic IP core used in the MACsec IP
are delivered as a part of a separate add-on installer. These models are not part of the
generic Intel Quartus Prime Pro Edition installer. Without the add-on installer, you can
compile your design using the Symmetric Cryptographic IP core, which is visible in the
Intel Quartus Prime IP Catalog but your simulation generates an error message.

To obtain the required add-on installer, contact Intel Support and quote case number
14015516629.

7.8. Running the Simulation

The design simulation for checking the Reference Design’s functionality is done in the
sim/ directory, which contains the necessary files for the simulation.

The <Project_Directory>/release)1p0/MACSEC_Release_1.0/
MACSEC_SysED_Instruction_Readme_1.0.txt file contains the steps to run the
simulation.

The steps for running the simulation are provided below.

1. Navigate to the env/ directory.
2. Source the environment settings. This is to be done only once in a shell.

Send Feedback MACsec Intel FPGA System Design User Guide

79
7. Generating the System Design
767516 | 2023.03.03

-- source setup.sh
3. For E-tile there is no need for support logic generation, but for F-tile designs follow
the steps below for supporting logic generation.
• Go to $SRD_ROOTDIR/sim/common and source the script below for the
appropriate support logic generation for F-tile based design.
a. Support logic generation 25G full design:
sh support_logic_gen.sh agx_nr_mudv 25G
b. Support logic generation 25G Lightweight MCDMA BFM design:
sh support_logic_gen.sh agx_nr_mudv 25G 1
c. Support logic generation 100G full design:
sh support_logic_gen.sh agx_nr_mudv 100G
d. Support logic generation 100G Lightweight MCDMA BFM design:
sh support_logic_gen.sh agx_nr_mudv 100G 1
4. For E-tile based design simulation: Go to $SRD_ROOTDIR/sim/common_ptile.
For F-tile based design simulation: Go to $SRD_ROOTDIR/sim/common.
Source the script generate_ip.sh, for IP simulation files generation.
The ip_list.f contains the list of all IPs used in the design.
sh generate_ip.sh ip_list.f
5. Navigate to the sim/runDir directory.
6. Run the simulation using the ./run_sim.sh command with arguments. The
supported arguments are shown in Simulation Command Arguments on page 73.
The test cases are provided in the sim/testcase folder. Each test case has the
test description mentioned in the first few lines of the file.

The following figure represents a typical run of the “combinedTrafficTest.sv” test:

TBINFO: address = 0a000034, readdata = 00000000

TBINFO: address = 0a000038, readdata = 00000000
TBINFO: address = 0a000058, readdata = 00000000
TBINFO: address = 0a000060, readdata = 00010000
TBINFO: address = 0a000064, readdata = 00000000
TBINFO: address = 0a000068, readdata = 00022df8
TBINFO: address = 0a00006c, readdata = 00000000
TBINFO: address = 0a000070, readdata = 000045bf
TBINFO: address = 0a000074, readdata = 00000000
TBINFO: address = 0a000078, readdata = 00004b72
TBINFO: address = 0a00007c, readdata = 00000000
TBINFO: address = 0a000000, readdata = 00000651
TBINFO: address = 0a000000, writedata32 = 00000611
TBINFO: Function 1 Clearing Packet Client Stats
TBINFO: address = 0a000000, readdata = 00000611
TBINFO: address = 0a000000, writedata32 = 00000699
TBINFO: address = 0a000000, readdata = 00000699
TBINFO: address = 0a000000, writedata32 = 00000611
TBINFO: Function 1 TX/RX packet check OK
*** TX PERFORMANCE MEASUREMENT *** no. of bytes = 0x22df8 num_ticks = 0x45bf
perf_data = 25.6000 Gb/s
*** RX PERFORMANCE MEASUREMENT *** no. of bytes = 0x10000 num_ticks = 0x4b72
perf_data = 10.8582 Gb/s
TBINFO: Function 1 Stop pkt gen TX
TBINFO: address = 0a000000, writedata32 = 00000100

MACsec Intel FPGA System Design User Guide Send Feedback

80
7. Generating the System Design
767516 | 2023.03.03

TBINFO: *****************************************
TBINFO: Simulation Passed.
TBINFO: Testbench complete
TBINFO: *****************************************

$finish called from file "../../tbTop/tb_macsec_srd.sv", line 125.

$finish at simulation time 522309564000

Following a successful simulation, a subdirectory is created in the /runDir directory

which corresponds to the test just run.

For example, the following directory is created for the “combinedTrafficTest.sv”

test:

<project directory>/sim/runDir/combinedTrafficTest

Within this test-specific directory, you can find all log files for the test as well as
a .vpd file (if you used the -m 1 simulation argument) which allows you to bring up
waves for the test run using the Synopsys DVE tool.

7.9. Building, Installing, and Running the Software

The building of the repository produces a set of application/drivers that need to be
loaded (or linked to) from the application. Perform the following steps to build and
install the software:
1. Log in as Super User.
Run the command below:
-- sudo -s
2. Install the dependent packages.
To Install the dependencies for drivers built on the platform Ubuntu 22.04-48-
generic, run the commands below:
Note: Make sure the kernel version is 5.15.0-48-generic. Also make sure that your
yum the repository proxy, the environment proxies are set properly, and you
have the sudo access.
-- sudo apt install -y build-essential libpython3-dev libdbus-1-dev
-- sudo apt-get -y install dbus libdbus-1-dev libdbus-glib-1-2 libdbus-
glib-1-dev
-- sudo apt-get install -y libnl-route-3-dev
-- sudo apt install libelf-dev
-- apt install openssh-server -y
-- sudo apt-get install libnl-genl-3-dev

3. Build all applications and drivers.

Run the command below:
-- ./build.sh all
a. Get the MCDMA module.
Run the command below:
-- git submodule update –init
b. Build all applications and drivers.

Send Feedback MACsec Intel FPGA System Design User Guide

81
7. Generating the System Design
767516 | 2023.03.03

Note: All drivers/applications are copied to the binaries folder. To build only
the required drivers/applications, refer to the app folders for patches
and build scripts.
Run the command below:
-- ./build.sh all
In addition to the above, you can do the following to build binaries separately
and they are copied to the binaries folder:
-- ./build.sh iptool
-- ./build.sh wpa_supplicant
-- ./build.sh linux_macsec
-- ./build.sh ipdriver
-- ./build.sh mcdma
c. Build individual applications and drivers.
i. To build the MCDSM driver, run the commands below, which build the
MCDM and the MACsec IP source code:
-- cd mcdma/driver/kmod/mcdma-netdev-driver/
-- make clean all
ii. To build the Linux MACsec driver, run the commands below:
-- cd linux_macsec/
-- ./linux_macsec_build.sh
iii. To build IProute2 – iptool application, run the commands below:
-- cd iptool/
-- ./iptool_build.sh
iv. To build the wpa_supplicant application, run the commands below:
-- cd wpa_supplicant/
-- ./build_wpa.sh
v. To build the CLI tool, run the commands below:
-- cd cli/
-- make clean all
vi. To build the MACsec IP driver code used for the HPS based design, run the
command below from the base folder to create the MACsec IP driver:
-- make clean all
d. Install the drivers.
i. MCDMA + MACsec IP driver.
Run the command below:
-- cd binaries

MACsec Intel FPGA System Design User Guide Send Feedback

82
7. Generating the System Design
767516 | 2023.03.03

Copy the pf_test.sh file to the binaries from the wpa_supplicant

directory. Run the pf_test.sh file to insert and create 2 name-spaces as
"ns0" and "ns1". Two MACsec IPs are present in the example design of the
HW. So 2 name-spaces are created.
Run the commands below to login to name-spaces:
-- ip netns exec ns0 bash
-- ip netns exec ns1 bash
ii. Linux MACsec driver.
After compiling/building, run the command below to insert the module:
-- insmod macsec.ko
iii. Run the CLI tool.
Run the command below to list the supported parameters:
-- ./cli_macsec -h
iv. Run the IP tool app.
The IP tool is an ip-macsec reference application.
Make sure the [Linux MACsec driver](#42-linux-macsec-driver) is inserted
before proceeding.
1. To configure the MACsec IP from iptool:
a. Run the command below to log in to "ns0":
-- ip netns exec ns0 bash
Below are some example `ip` commands to configure `ifc_mcdma0`
for the `macsec0` interface:

-- ./ip link add link ifc_mcdma0 macsec0 type macsec address

6e:ed:32:da:fe:4b port 0 encrypt off offload mac
-- ./ip macsec add macsec0 tx sa 0 sc 0 on pn 1 key 01
ABCD1234567891011121314151617181
-- ./ip macsec add macsec0 rx port 0 address 96:2d:5a:25:ac:ae on
-- ./ip macsec add macsec0 rx address 96:2d:5a:25:ac:ae port 0 sa
0 sc 0 pn 1 on key 01 ABCD1234567891011121314151617181

b. Run the command below to log in to "ns1":

-- ip netns exec ns1 bash
Below are some example `ip` commands to configure `ifc_mcdma1`
for the `macsec1` interface:

-- ./ip link add link ifc_mcdma1 macsec0 type macsec address

96:2d:5a:25:ac:ae port 0 encrypt off offload mac
-- ./ip macsec add macsec0 tx sa 0 sc 0 on pn 1 key 01
ABCD1234567891011121314151617181
-- ./ip macsec add macsec0 rx port 0 address 6e:ed:32:da:fe:4b on
-- ./ip macsec add macsec0 rx address 6e:ed:32:da:fe:4b port 0 sa
0 sc 0 pn 1 on key 01 ABCD1234567891011121314151617181

2. Follow these steps to initiate traffic via the CLI tool after configuring
using the ip tool:

Send Feedback MACsec Intel FPGA System Design User Guide

83
7. Generating the System Design
767516 | 2023.03.03

a. Run the command below to check the HSSI status (Read data
should be 0x1E):
--./cli_macsec ppbb_reg_read -d ifc_mcdma0 -i 0 -o
0x0054 -r
b. Run the command below to configure the number of packets to be
transmitted:
--./cli_macsec ppbb_reg_write -d ifc_mcdma0 -i 0 -o
0x001c -w 0x00000400
c. Run the command below to configure the packet size (min size in
[13:0], max size in [29:16]):
--./cli_macsec ppbb_reg_write -d ifc_mcdma0 -i 0 -o
0x0020 -w 0x00500050
d. Run the command below to start traffic in the dynamic mode:
--./cli_macsec ppbb_reg_write -d ifc_mcdma0 -i 0 -o
0x0000 -w 0x00008611
e. Run the command below to read the checker packet counter (wait
for this counter to display equal packet number before proceeding to
the next step):
--./cli_macsec ppbb_reg_read -d ifc_mcdma0 -i 0 -o
0x005c -r
3. To show the MACsec IP statistics, run the command below to login to
the specific console, i.e. ns0 or ns1, to read out the statistics:
--./ip -s macsec show
v. Run the wpa_supplicant.
The WPA is a MACsec MKA reference application.
Follow these steps below:
1. Insert the [McDMA+MACsec IP drivers](#41-mcdma--macsec-ip-
driver).
2. Insert the [Linux MACsec driver](#42-linux-macsec-driver).
3. Run the wpa_supplicant.
Below is an example:
--./wpa_supplicant -i ifc_mcdma0 -Dmacsec_linux -c
mka_default.conf
Below are example steps to evaluate the wpa_supplicant, which needs
three terminals in parallel.
a. In the terminal window 1, do the following:
a. Navigate to binaries folder.
b. Log in to the ns0: ip netns exec ns0 bash.
c. Execute this command: sh wpa_offload.sh.
b. In the terminal window 2, do the following:
a. Navigate to binaries folder.

MACsec Intel FPGA System Design User Guide Send Feedback

84
7. Generating the System Design
767516 | 2023.03.03

b. Log in to the ns1: ip netns exec ns1 bash.

c. Execute this command: sh wpa_offload.sh1-+.
c. In the terminal window 3, do the following:
a. Navigate to binaries folder.
b. Log in to the ns0: ip netns exec ns0 bash.
c. Run the steps as in step IV-2 (initiate traffic via the CLI tool
after configuring using the ip tool) above.
For statistics, repeat step IV-3 (show the MACsec IP statistics)
above.

Send Feedback MACsec Intel FPGA System Design User Guide

85
767516 | 2023.03.03

Send Feedback

8. Document Revision History for MACsec System Design

User Guide
Document Version Intel® Quartus® Hardware Reference Software Reference Changes
Prime Version Design Version Design Version

2023.03.03 22.4 1.0.0 1.0.0 Initial release.

Intel Corporation. All rights reserved. Intel, the Intel logo, and other Intel marks are trademarks of Intel
Corporation or its subsidiaries. Intel warrants performance of its FPGA and semiconductor products to current
specifications in accordance with Intel's standard warranty, but reserves the right to make changes to any ISO
products and services at any time without notice. Intel assumes no responsibility or liability arising out of the 9001:2015
application or use of any information, product, or service described herein except as expressly agreed to in Registered
writing by Intel. Intel customers are advised to obtain the latest version of device specifications before relying
on any published information and before placing orders for products or services.
*Other names and brands may be claimed as the property of others.