DriveFuzz

Prerequisite

• Strictly required: Ubuntu 18.04 (requirement of Autoware and ROS melodic)

• Python 3

• Python dependencies

  $ pip3 install --user pygame scikit-fuzzy docker

• setuptools needs to be downgraded due to a bug in CARLA.

  $ pip3 install -Iv setuptools==47.3.1

Setting up docker and nvidia driver

• Install docker-ce

1. Remove existing installation, add GPG key, and install docker-ce

   $ sudo apt remove docker docker-engine docker.io
                     $ sudo apt update
                     $ sudo apt install apt-transport-https ca-certificates curl software-properties-common
                     $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
                     $ sudo apt-key fingerprint 0EBFCD88
                     $ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs)stable"
                     $ sudo apt update
                     $ sudo apt install docker-ce

2. Check installation. The following should print a message starting with "Hello from Docker!"

   $ sudo docker run hello-world

• Set up permissions

1. Create group docker and add yourself to it

   $ sudo groupadd docker
                     $ sudo usermod -aG docker ${USER}

2. Log out and log back in

   $ sudo su - ${USER}

3. Try running docker without root permission this time

   $ docker run hello-world

• Make sure you have nvidia driver on Host

1. Search for available drivers

   $ sudo add-apt-repository ppa:graphics-drivers
                     $ sudo apt update
                     $ ubuntu-drivers devices

2. Install one of the compatible drivers. nvidia-driver-415 worked for me.

   $ sudo apt install nvidia-driver-415
                     $ sudo reboot now
                     $ nvidia-smi

• Install nvidia-docker2

1. Install

   $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
                     $ curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
                     $ curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
                     
                     $ sudo apt update
                     $ sudo apt install nvidia-docker2
                     $ sudo systemctl restart docker

2. Test the installation

   $ docker run --runtime=nvidia --rm nvidia/cuda:10.0-base nvidia-smi

Install CARLA 0.9.10.1 docker

• Pull docker image

  $ docker pull carlasim/carla:0.9.10.1

• Quick-running CARLA CARLA can be run using a wrapper script run_carla.sh. If you have multiple GPUs installed, it is recommended that you "pin" CARLA simulator to one of the GPUs (other than #0). You can do that by opening run_carla.sh and modifying the following:

  -e NVIDIA_VISIBLE_DEVICES={DESIRED_GPU_ID} --gpus 'device={DESIRED_GPU_ID}

• To run carla simulator, execute the script:

  $ ./run_carla.sh

• It will run carla simulator container, and name it carla-${USER} .

  To stop the container, do:

  $ docker rm -f carla-${USE}

• Building CARLA API

  $ cd carla
                    $ make PythonAPI

  Ensure that PythonAPI/carla/dist/carla-0.9.10-py3.x-linux-x86_64.egg has been created after building.

  Return to the project root directory.

Building carla-autoware docker

• Get additional files

  $ cd carla-autoware
                    $ git clone https://bitbucket.org/carla-simulator/autoware-contents.git

• Build carla-autoware docker

  $ ./build.sh

Installing ROS-melodic

• ROS is required on the host in order for DriveFuzz to communicate with the Autoware container.

  $ sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
                    $ sudo apt install curl
                    $ curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | sudo apt-key add -
                    $ sudo apt update
                    $ sudo apt install ros-melodic-desktop-full
                    
                    $ source /opt/ros/melodic/setup.bash

Run DriveFuzz

• Clone this repository to ~/drivefuzz

  $ git clone https://gitlab.com/s3lab-code/public/drivefuzz.git ~/drivefuzz

• Run carla simulator

  First check if carla is already running:

  $ docker ps | grep carla-$USER

  If not, run CARLA container by executing:

  $ cd ~/drivefuzz
                $ ./run_carla.sh

• Prepare environment

  $ cd ~/drivefuzz
                $ ./init.sh
                $ source /opt/ros/melodic/setup.bash

• Run fuzzing

1. Testing Behavior Agent

   $ cd ~/drivefuzz/src
                   $ ./fuzzer.py -o out-artifact -s seed-artifact -c 5 -m 5 -t behavior --timeout 60 --town 1 --strategy all

   • Outputs (bugs, mutated test inputs, metadata, ...) will be stored in out-artifact
   • Sample seeds are provided in seed-artifact
   • Adjust the number of cycles (-c) and size of mutated population (-m) (please refer to Algorithm 1 in the paper)
   • You can try other mutation strategies (congestion, entropy, instability, trajectory); (please refer to Section 4.2.4 in the paper)
   • Check ./fuzzer.py --help for further instructions

   Please note that Behavior Agent has a bug in the controller (that we found and reported), which leads to frequent lane invasions and finishing before reaching the goal. You can prevent these from triggering the misbehavior detector by adding --no-lane-check and --no-other-check flags, e.g.,

   $ ./fuzzer.py -o out-artifact -s seed-artifact -c 5 -m 5 -t behavior --timeout 60 --town 1 --strategy all --no-lane-check --no-other-check

2. Testing Autoware

   $ cd ~/drivefuzz/src
                   $ rm -rf out-artifact
                   $ ./fuzzer.py -o out-artifact -s seed-artifact -c 5 -m 5 -t autoware --timeout 60 --town 1 --strategy all --sim-port 2000