This study applies machine learning techniques and novel statistical features for two important classification problems in secure computing: malware detection and file fragment type identification. We observe combinations of information-theoretic and Natural Language Processing features extracted from byte level file content. To the extent possible, we replicate recent studies to validate the use of these features and expand on recent work by combining features from malware to detection to fragment identification tasks and vice versa. By avoiding the use of extracted file signatures and strings, this study contributes techniques that may be more resistant to obfuscation attacks, lead to enhanced prediction rates for zero-day malware files, and improved forensics on broken fragments where file metadata information is not available. We evaluate our results against recent works and report the highest performing algorithms and combinations of features for each task.

Subjects: Machine Learning; Information Theory; File Forensics; Malware Detection; Digital Fingerprinting; Anomaly Detection