AI: Anomaly Detection in logfiles - Revision history

ARessl at 15:46, 27 July 2022

2022-07-27T15:46:30Z

← Older revision		Revision as of 15:46, 27 July 2022
Line 137:		Line 137:

	* [[Jetson AGX Xavier Development Kit]]		* [[Jetson AGX Xavier Development Kit]]

			== References ==
			* Dataset https://www.unb.ca/cic/datasets/ids-2018.html
			* learndatasci Guide https://www.learndatasci.com/glossary/binary-classification/
			* Binary Classification with keras Guide https://machinelearningmastery.com/binary-classification-tutorial-with-the-keras-deep-learning-library/


	[[Category:Documentation]]		[[Category:Documentation]]

2022-07-13T14:54:20Z

← Older revision		Revision as of 14:54, 13 July 2022
Line 1:		Line 1:
	~~<div style="border: 1px solid #8a6d3b; background-color: #fcf8e3; color: #8a6d3b; padding: 5px 10px; margin-bottom: 5px; text-align: justify">~~
	~~&emsp;➤ IMPORTANT: This page is still under construction.</div>~~
	== Summary ==		== Summary ==

2022-07-13T14:53:36Z

← Older revision		Revision as of 14:53, 13 July 2022
Line 10:		Line 10:
	* Packages: TensorFlow, Keras, Pandas, sklearn, numpy, seaborn, matplotlib		* Packages: TensorFlow, Keras, Pandas, sklearn, numpy, seaborn, matplotlib
	* Software: Pycharm or any other python editor		* Software: Pycharm or any other python editor
	* Dataset: We used the following ~~Dataset [~~[https://www.unb.ca/cic/datasets/ids-2018.html]]		* Dataset: We used the following [https://www.unb.ca/cic/datasets/ids-2018.html Dataset]

	== Description ==		== Description ==

2022-07-13T14:52:42Z

← Older revision		Revision as of 14:52, 13 July 2022
Line 10:		Line 10:
	* Packages: TensorFlow, Keras, Pandas, sklearn, numpy, seaborn, matplotlib		* Packages: TensorFlow, Keras, Pandas, sklearn, numpy, seaborn, matplotlib
	* Software: Pycharm or any other python editor		* Software: Pycharm or any other python editor
	* Dataset: We used the following Dataset https://www.unb.ca/cic/datasets/ids-2018.html		* Dataset: We used the following Dataset [[https://www.unb.ca/cic/datasets/ids-2018.html]]

	== Description ==		== Description ==

2022-07-13T14:51:37Z

Requirements

← Older revision		Revision as of 14:51, 13 July 2022
Line 10:		Line 10:
	* Packages: TensorFlow, Keras, Pandas, sklearn, numpy, seaborn, matplotlib		* Packages: TensorFlow, Keras, Pandas, sklearn, numpy, seaborn, matplotlib
	* Software: Pycharm or any other python editor		* Software: Pycharm or any other python editor
			* Dataset: We used the following Dataset https://www.unb.ca/cic/datasets/ids-2018.html

	== Description ==		== Description ==

2022-07-13T14:49:40Z

← Older revision		Revision as of 14:49, 13 July 2022
Line 35:		Line 35:
	logfile_features.dropna(inplace=True)		logfile_features.dropna(inplace=True)
	Our dataset has labels which define if its an attack or not, so we replace them with numericals (0 and 1)		Our dataset has labels which define if its an attack or not, so we replace them with numericals (0 and 1)

	In our case Benign is for standard user traffic and DoS attacks-Slowloris and DoS attacks-GoldenEye is for traffic where a DoS attack occured		In our case Benign is for standard user traffic and DoS attacks-Slowloris and DoS attacks-GoldenEye is for traffic where a DoS attack occured
	logfile_features["Label"].replace({"Benign": 0, "DoS attacks-Slowloris": 1, "DoS attacks-GoldenEye": 1}, inplace=True)		logfile_features["Label"].replace({"Benign": 0, "DoS attacks-Slowloris": 1, "DoS attacks-GoldenEye": 1}, inplace=True)

2022-07-13T14:49:27Z

Step 1 - Read the dataset

← Older revision		Revision as of 14:49, 13 July 2022
Line 35:		Line 35:
	logfile_features.dropna(inplace=True)		logfile_features.dropna(inplace=True)
	Our dataset has labels which define if its an attack or not, so we replace them with numericals (0 and 1)		Our dataset has labels which define if its an attack or not, so we replace them with numericals (0 and 1)
			In our case Benign is for standard user traffic and DoS attacks-Slowloris and DoS attacks-GoldenEye is for traffic where a DoS attack occured
	logfile_features["Label"].replace({"Benign": 0, "DoS attacks-Slowloris": 1, "DoS attacks-GoldenEye": 1}, inplace=True)		logfile_features["Label"].replace({"Benign": 0, "DoS attacks-Slowloris": 1, "DoS attacks-GoldenEye": 1}, inplace=True)
	Next we shuffle our dataset		Next we shuffle our dataset
Line 60:		Line 61:
	normed_test_data = norm(test_dataset, train_stats)		normed_test_data = norm(test_dataset, train_stats)
	normed_valid_dataset = norm(valid_dataset, train_stats)		normed_valid_dataset = norm(valid_dataset, train_stats)


	=== Step 2 - Create a model ===		=== Step 2 - Create a model ===

2022-07-13T14:48:09Z

Step 4 - Plot the results

← Older revision		Revision as of 14:48, 13 July 2022
Line 130:		Line 130:
	ax.yaxis.set_ticklabels(['No Attack', 'Attack'])		ax.yaxis.set_ticklabels(['No Attack', 'Attack'])
	plt.show()		plt.show()

			[[File:AI_model_accuracy.png\|500px\|Model Accuracy]]
			[[File:AI_confusion_matrix.png\|500px\|Confusion Matrix]]

	== Used Hardware ==		== Used Hardware ==

2022-07-13T14:47:09Z

Step 2 - Create a model

← Older revision		Revision as of 14:47, 13 July 2022
Line 81:		Line 81:
	optimizer = tf.optimizers.Adam(learning_rate)		optimizer = tf.optimizers.Adam(learning_rate)

	Lastly we need to compile the model, for the loss function we use ~~BinaryCorssentropy~~, our optimizer and the metric should be the accuarcy of the model		Lastly we need to compile the model, for the loss function we use BinaryCrossentropy, our optimizer and the metric should be the accuarcy of the model
	model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),		model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
	optimizer=optimizer,		optimizer=optimizer,
	metrics=['accuracy'])		metrics=['accuracy'])


	=== Step 3 - Train the model ===		=== Step 3 - Train the model ===

2022-07-13T14:44:07Z

Step 0 - Import the needed packages/libraries

← Older revision		Revision as of 14:44, 13 July 2022
Line 17:		Line 17:
	from keras.callbacks import EarlyStopping, ModelCheckpoint # for training		from keras.callbacks import EarlyStopping, ModelCheckpoint # for training
	from keras.models import Sequential, load_model # for model		from keras.models import Sequential, load_model # for model
	from keras.layers import Dense # layers		from keras.layers import Dense, Activation # layers and activation function
	import pandas as pd # read/prep dataset		import pandas as pd # read/prep dataset
	pd.options.mode.chained_assignment = None # removes warning		pd.options.mode.chained_assignment = None # removes warning