ABSTRACT

Control and server programs running on IoT gateways and embedded devices are still open to control-oriented and data-oriented attacks as traditional heavy-duty software countermeasures cannot provide defenses on these devices with limited computing capability, memory, and storage space and possibly in absence of an underlying operating system. Existing host-based program anomaly detection techniques primarily focus on subsets of applications, e.g., on system calls or calls to predefined libraries. They are either not deployable on gateway and embedded devices or are insufficient to detect subtle control-oriented attacks that introduce new anomalous call relationships at the application level. Moreover, they cannot detect malicious intra- and inter-procedural program behavior induced by subtle data-oriented attacks as they fail to capture fine-grained control-flow transfers (e.g., branches or loops) for modeling short-range executions while simultaneously preserving order-sensitivity in long-range behavioral modeling. This dissertation addresses several challenges in reasoning about program execution using fine-grained control-flow transfers and in tracing attributes using software-based low overhead instrumentation. 

In first part of the dissertation, we present LANCET, a system for lightweight host-based intrusion detection on IoT gateways. Our modeling captures inter-procedural attributes using all plausible caller-callee relationships in an application. In the second study, we present EDISON, a framework for detecting data-oriented attacks on embedded IoT devices using deep learning. Our modeling integrates two levels to capture both intra- and inter-procedural attributes using graph- and language-based modeling techniques, respectively. Prototypes of our approaches are implemented, and their effectiveness is demonstrated by detecting all reproduced real-world and crafted attacks with relatively high accuracy and low false alarm rates, comparing them against the closest state-of-the-art techniques. We show that they incur low average binary size overhead due to instrumentation and low runtime overhead due to tracing, while detecting attacks promptly. We validate the model designs through ablation studies and verify robustness to adversaries.