Hardware-In-the-Loop (HIL) network simulation for supporting electrical power grid sub-stations

This is the ns-3 applications from my master thesis conducted in the spring of 2021. The master thesis developed a validated verison of DPDK-enhanced ns-3 capable of supporting multiple NIC. Follow this HowTo to replay the thesis work.

Alterations to the ns-3.33 source code are made in the folder “/ns-3.33/src/fd-net-device/” to:

dpdk-net-device.cc/.h

dpdk-net-device-helper.cc/.h

fd-net-device.cc/.h

fd-net-device-helper.cc/.h

HowTo guide for running the thesis simulations

Setting the Ubuntu Environment

The OS used in the thesis was Ubuntu 20.04.

1. First download this repository to your Ubuntu computer. It contain altered ns-3.33.

   git clone https://gitlab.stud.idi.ntnu.no/johnnymo/masterthesishil.git

2. Install DPDK by using the apt-get command below.

   • DPDK version used in the thesis was 20.11, so to prevent any compatibility issue, I recomend using this version.
   • DPDK can manually be downloaded from Downloads
   • Check that your NIC is supported by DPDK Supported HW

   sudo apt-get install dpdk

• Make sure you can run the following from the commandline terminal:

  sudo dpdk-devbind.py -s

• If not, add the dpdk commands to path.

3. Alter the GRUB bootloader configuration file to enable hugepages and set driver parameters. This is done by changing the GRUB_CMDLINE_LINUX_DEFAULT parameter in the file “/etc/default/grub”. The provided parameter was my entire parameter setting.

   • "maybe-ubiquity" was present by default.
   • "default_hugepagesz=1G hugepagesz=1G hugepages=1" set 1GB hugepage by default and enables 1 hugepage. Tests showed that the application only used 1 hugepage. Set more if needed, but the more you set the more memory will be unavailable by your OS when mounting (see step 5).
   • "iommu=pt iommu=on intel_iommu=on" are for enabling the vfio-pci (and uio_igb) driver that was used by DPDK in the thesis.
   • "isolcpus=4,10,5,11" isolate logical cpu cores and prevents the OS from using the logical cores. Add the cores you would like to use in your application here. The thesis used logical cores 4, 5, 10 and 11 that was used to run the DPDK application on.
     • Use lscpu or lstopo to see how many logical cpu cores your computer have.

   GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity default_hugepagesz=1G hugepagesz=1G hugepages=1 iommu=pt iommu=on intel_iommu=on isolcpus=4,10,5,11"

4. Update GRUB.

   sudo update-grub

   OR

   sudo update-grub2

5. Create a folder you will mount your hugepage(s) on.

   mkdir /mnt/huge

6. Mount hugepages by default on startup by adding the following line at the bottom of the file “/etc/fstab”.

   nodev /mnt/huge hugetlbfs defaults 0 0

7. Restart the computer.

Build ns-3

NOTE: You must have a c++ compiler (gcc used in thesis) and python version >=3.5.

1. Go to folder /ns-3.33/.

   cd ns-3.33

2. Configure the ns3 waf builder.

   sudo ./waf configure --build-profile=optimized --enable-examples --enable-tests --enable-sudo

3. Build ns-3 by running:

   sudo ./waf 

• Hopefully ns-3 is able to be built. If not, please refer to ns-3 Installation guide

4. Try to run a sample ns-3 application with:

   sudo ./waf --run first

Running DpdkOneNode

The examples from the thesis are located in /ns-3.33/scratch/

The simulation used in the thesis was dpdkOneNode.cc

1. To run dpdkOneNode some parameters might need to be changed:

• dpdkDriver: Set which driver to use with DPDK.

• dev[]: Array of strings that contain the PCI-address(es) of your NIC(s) to be used in the simulation.

  • You can use the following command in the terminal to get more info about the NIC's and their PCI addresses.

    dpdk-devbind.py -s

• lCorelist: Should be the same as you chose to isolate in the GRUB parameter isolcpus as it tells the DPDK what cores to use in te simulation

• firstLCore: Sets the logical core of the first NIC you would like to handle incoming traffic on. Must be part of lCorelist.

• secLCore: Sets the logical core of the second NIC you would like to handle incoming traffic on. Must be part of lCorelist.

2. The dpdkOneNode accept input parameter:

• pcap: Boolean if you want to write pcap files from the node in the simulation to file
• routeTable Boolean if you want to write routing table of the node to file.
• runTime: Sets how long you would like the simulation to run in seconds.

3. Example of running dpdkOneNode for 30 seconds, writing both pcap and routing tables to file:

   sudo ./waf --run "dpdkOneNode --runTime=30 --pcap --routeTable"

4. For running FdNetDev and TapBridge used in the thesis use the following commands. (Examples with all CLI parameters)

   sudo ./waf --run "emuFdSwitchTopo --runTime=10 --int0=eth0 --int0=eth1 --pcap --routeTable"

   OR

   sudo ./waf --run "tapSwitch --pcap --runTime=10" 

   • The configuration of the environment is explained in the files /ns-3.33/scratch/emuFdOneNode.cc and /ns-3.33/scratch/tapSwitch.cc respectively.