In this paper, we address the problem of detecting misbehaving vehicles in Vehicular Ad Hoc Network (VANET) using Quality of Service Optimized Link State Routing (QoS-OLSR) protocol. According to this protocol, vehicles might misbehave either during the clusters’ formation by claiming bogus information or after clusters are formed. A vehicle is considered as selfish or misbehaving once it over-speeds the maximum speed limit or under-speeds the minimum speed limit where such a behavior will lead to a disconnected network. As a solution, we propose a two-phase model that is able to motivate nodes to behave cooperatively during clusters’ formation and detect misbehaving nodes after clusters are formed. Incentives are given in the form of reputation and linked to network’s services to motivate vehicles to behave cooperatively during the first phase. Misbehaving vehicles can still benefit from network’s services by behaving normally during the clusters’ formation and misbehave after clusters are formed. To detect misbehaving vehicles, cooperative watchdog model based on Dempster–Shafer is modeled where evidences are aggregated and cooperative decision is made. Simulation results show that the proposed detection model is able to increase the probability of detection, decrease the false negatives, and reduce the percentage of selfish nodes in the vehicular network, while maintaining the Quality of Service and stability.
