README.md

# Hardware-In-the-Loop (HIL) network simulation for supporting electrical power grid sub-stations

This is the ns-3 applications from my master thesis conducted in the spring of 2021. The master thesis developed a validated verison of DPDK-enhanced ns-3 capable of supporting multiple NIC. Follow this HowTo to replay the thesis work.

____
Alterations to the ns-3.33 source code are made in the folder “/ns-3.33/src/fd-net-device/” to:

dpdk-net-device.cc/.h

dpdk-net-device-helper.cc/.h

fd-net-device.cc/.h

fd-net-device-helper.cc/.h

____
## HowTo guide for running the thesis simulations

### Setting the Ubuntu Environment

The OS used in the thesis was Ubuntu 20.04.

1. First download this repository to your Ubuntu computer. It contain altered ns-3.33.

    ```sh
    git clone https://gitlab.stud.idi.ntnu.no/johnnymo/masterthesishil.git
    ```

2. Install DPDK by using the apt-get command below.
    * DPDK version used in the thesis was 20.11, so to prevent any compatibility issue, I recomend using this version.
    * DPDK can manually be downloaded from [Downloads](https://core.dpdk.org/download/)
    * Check that your NIC is supported by DPDK [Supported HW](https://core.dpdk.org/supported/)

    ```sh
    sudo apt-get install dpdk
    ```

* Make sure you can run the following from the commandline terminal:

    ```sh
    sudo dpdk-devbind.py -s
    ```

* If not, add the dpdk commands to path.

3. Alter the GRUB bootloader configuration file to enable hugepages and set driver parameters. This is done by changing the GRUB_CMDLINE_LINUX_DEFAULT parameter in the file “/etc/default/grub”. The provided parameter was my entire parameter setting.
    * ```"maybe-ubiquity"``` was present by default.
    * ```"default_hugepagesz=1G hugepagesz=1G hugepages=1"``` set 1GB hugepage by default and enables 1 hugepage. Tests showed that the application only used 1 hugepage. Set more if needed, but the more you set the more memory will be unavailable by your OS when mounting (see step 5).
    * ```"iommu=pt iommu=on intel_iommu=on"``` are for enabling the vfio-pci (and uio_igb) driver that was used by DPDK in the thesis.
    * ```"isolcpus=4,10,5,11"``` isolate logical cpu cores and prevents the OS from using the logical cores. Add the cores you would like to use in your application here. The thesis used logical cores 4, 5, 10 and 11 that was used to run the DPDK application on.
        * Use ```lscpu``` or ```lstopo``` to see how many logical cpu cores your computer have. 

    ```sh
    GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity default_hugepagesz=1G hugepagesz=1G hugepages=1 iommu=pt iommu=on intel_iommu=on isolcpus=4,10,5,11"
    ```

4. Update GRUB.

    ```sh
    sudo update-grub
    ```
    OR

    ```sh
    sudo update-grub2
    ```


5. Create a folder you will mount your hugepage(s) on.

    ```sh
    mkdir /mnt/huge
    ```

6. Mount hugepages by default on startup by adding the following line at the bottom of the file “/etc/fstab”.

    ```sh
    nodev /mnt/huge hugetlbfs defaults 0 0
    ```

7. Restart the computer.

### Build ns-3

**NOTE:** You must have a c++ compiler (gcc used in thesis) and python version >=3.5.

1. Go to folder /ns-3.33/.
    ```sh
    cd ns-3.33
    ```

2. Configure the ns3 *waf* builder.

    ```sh
    sudo ./waf configure --build-profile=optimized --enable-examples --enable-tests --enable-sudo
    ```

3. Build ns-3 by running:

    ```sh
    sudo ./waf 
    ```

* Hopefully ns-3 is able to be built. If not, please refer to [ns-3 Installation guide](https://www.nsnam.org/docs/release/3.33/tutorial/html/getting-started.html)

4. Try to run a sample ns-3 application with:

    ```sh
    sudo ./waf --run first
    ```

### Running DpdkOneNode

**The examples from the thesis are located in /ns-3.33/scratch/**

The simulation used in the thesis was ```dpdkOneNode.cc```

1. To run dpdkOneNode some parameters might need to be changed:

* ```dpdkDriver```: Set which driver to use with DPDK.
* ```dev[]```: Array of strings that contain the PCI-address(es) of your NIC(s) to be used in the simulation.

    * You can use the following command in the terminal to get more info about the NIC's and their PCI addresses.

        ```sh
        dpdk-devbind.py -s
        ```

* ```lCorelist```: Should be the same as you chose to isolate in the GRUB parameter *isolcpus* as it tells the DPDK what cores to use in te simulation
* ```firstLCore```: Sets the logical core of the first NIC you would like to handle incoming traffic on. Must be part of lCorelist.
* ```secLCore```: Sets the logical core of the second NIC you would like to handle incoming traffic on. Must be part of lCorelist.

2. The *dpdkOneNode* accept input parameter:

* ```pcap```: Boolean if you want to write pcap files from the node in the simulation to file
* ```routeTable``` Boolean if you want to write routing table of the node to file.
* ```runTime```: Sets how long you would like the simulation to run in seconds.

3. Example of running dpdkOneNode for 30 seconds, writing both pcap and routing tables to file:

    ```sh
    sudo ./waf --run "dpdkOneNode --runTime=30 --pcap --routeTable"
    ```

4. For running FdNetDev and TapBridge used in the thesis use the following commands. (Examples with all CLI parameters)

    ```sh
    sudo ./waf --run "emuFdSwitchTopo --runTime=10 --int0=eth0 --int0=eth1 --pcap --routeTable"
    ```
    OR
    ```sh
    sudo ./waf --run "tapSwitch --pcap --runTime=10" 
    ```
    * The configuration of the environment is explained in the files /ns-3.33/scratch/emuFdOneNode.cc and /ns-3.33/scratch/tapSwitch.cc respectively.