Determining whether a Javascript sample is malicious is not computable (https://en.wikipedia.org/wiki/Computable_function) : we are looking for an algorithm that takes a program (which can be seen as an arbitrary Turing Machine) as an input and whose output is a property of the execution of that program.
If you are unfamiliar with the theory of computability and want ot get an intuitive sense of this, imagine writing a JS sample that non trivially never terminates. A simple while(1){} would not do the trick because it can be trivially proven (without executing it) that it never terminates.
A program terminating depending on the answer to some complex mathematical problem (e.g. finding whether a big number is prime) can not be proven to terminate short of actually solving the problem, the best method for doing so being to actually execute the program.
Therefore, the best way to now if this program will terminate is to execute it, which may never ends. That is why deciding a property about the execution of that program is not computable in the general case.
This does not deter us from trying though, because in practice a program that does not terminate in a few seconds will be interrupted by the browser, and is therefore neither malicious nor begnin, it is non-fonctional. The goal here is to devise some indicator of malignity of a JS sample without even executing it (who wants to execute malicious code ?).
\cite{likarish2009obfuscated}. Bonne intro, bon blabla, mais ils ont créé un détecteur d'obfuscation plus qu'autre chose. On utilise quand même leur features.
On se limite aux features qu'on peut calculer sans même parser le JS (ne fut-ce que parce qu'on est pas à l'abri d'une attaque sur le parser.
In [48]:
import glob
import string
import re
import numpy as np
# Loading the data
data = []
for js_file in glob.glob('Javascript/*/*'):
new = {}
new['name'] = js_file.split('/')[-1]
new['code'] = open(js_file,'r').read()
if new['name'][-2:] == 'js':
if new['name'][-6:] == 'min.js':
new['nature'] = 'Minified'
new['color'] = 'b'
else:
new['nature'] = 'Normal'
new['color'] = 'g'
elif new['name'][-3:] == 'out':
new['nature'] = 'Malicious'
new['color'] = 'r'
data.append(new)
In [49]:
def length(code):
return len(code)
def nb_lines(code):
return len(code.split('\n'))
def avg_char_per_line(code):
return length(code)/nb_lines(code)
def nb_strings(code):
'''Ugly approximation, no simple way out of this short of actually parsing the JS.'''
return len(code.split("'"))+len(code.split('"'))
def nb_non_printable(code):
'''\cite{likarish2009obfuscated} use unicode symbol, but we are more general'''
return len([x for x in code if not x in string.printable])
hex_octal_re = re.compile('([^A-F0-9]0[0-7]+|0x[A-F0-9]+)')
def hex_or_octal(code):
'''Ugly as hell, but we dont want to parse'''
return len(list(hex_octal_re.finditer(code)))
def max_nesting_level(code):
l = 0
max_l = 0
for c in code:
if c in '({[':
l+=1
max_l = l if l > max_l else max_l
elif c in ')}]':
l-=1
return max_l
features = [length, nb_lines, avg_char_per_line, nb_strings, nb_non_printable, hex_or_octal, max_nesting_level]
In [51]:
X = np.array([[f(x['code']) for f in features] for x in data])
X[:30]
Out[51]:
In [57]:
#http://scikit-learn.org/stable/auto_examples/manifold/plot_compare_methods.html#example-manifold-plot-compare-methods-py
from sklearn import manifold
%matplotlib inline
import matplotlib.pylab as plt
n_neighbors = 10
n_components = 2
#Y = manifold.Isomap(n_neighbors, n_components).fit_transform(X)
#Y = manifold.LocallyLinearEmbedding(n_neighbors, n_components,
# eigen_solver='auto').fit_transform(X)
Y = manifold.MDS(n_components, max_iter=100, n_init=1).fit_transform(X)
#Y = manifold.SpectralEmbedding(n_components=n_components,
# n_neighbors=n_neighbors).fit_transform(X)
#Y = manifold.TSNE(n_components=n_components, init='pca', random_state=0).fit_transform(X)
plt.scatter(Y[:, 0], Y[:, 1], c=[x['color'] for x in data], alpha=0.2)
for label, x, y in zip([x['name'] for x in data], Y[:, 0], Y[:, 1]):
if '.js' in label and not ('min.' in lab2el):
plt.annotate(label ,xy=[x,y])
plt.savefig('toto.pdf')
In [42]:
[[x['name'],hex_or_octal(x['code'])] for x in data[:3]]
Out[42]:
In [43]:
[x for x in data[-3]['code'] if not x in string.printable]
Out[43]:
On arrive assez facilement à différencier les fichiers JS extrêmement gentils de ceux un peu moins sympa. Le problème est qu'il y a des fichiers pas sympa et des fichiers innocents mélangés au même endroit.
On va voir de quoi il retourne en enlevant de l'analyse les fichiers trop sympas et facile à détecter, ce qui nous laissera nous concentrer sur les fichiers problématiques.
Pour ça on crée le répertoire 'hard_js' et le notebook correspondant.