CVE Webscraper - Revision history

LLaschitz at 10:24, 13 July 2022

2022-07-13T10:24:33Z

← Older revision		Revision as of 10:24, 13 July 2022
Line 42:		Line 42:

	=== Use ===		=== Use ===

			For a detailed explation of how to setup and use the software please refer to the README.md at the [https://git.fh-campuswien.ac.at/sisch/cve-crawlwe FH Campus Wien Git] or at [https://github.com/karaxis/wpf1/tree/main Github].

	==Sources==		==Sources==

LLaschitz: /* General */

2022-07-13T10:22:52Z

General

← Older revision		Revision as of 10:22, 13 July 2022
Line 34:		Line 34:
	Quickly find known vulnerabilities for a certain application.		Quickly find known vulnerabilities for a certain application.
	Runs on windows and Linux.		Runs on windows and Linux.

			The application was written in C# with the [https://dotnet.microsoft.com/en-us/ .NET 6.0 Framework] with a focus on a crossplatform availability.

			The software is available at the [https://git.fh-campuswien.ac.at/sisch/cve-crawlwe FH Campus Wien Git] or at [https://github.com/karaxis/wpf1/tree/main Github].

			It scans the vulnerability database of [https://cve.mitre.org/ mitre.org] for vulerabilities for a given software and saves all CVE numbers with corresponding links in a file.

	=== Use ===		=== Use ===

LLaschitz: LLaschitz moved page (Article title) to CVE Webscraper: Title was Article Title

2022-07-13T10:10:51Z

LLaschitz moved page (Article title) to CVE Webscraper: Title was Article Title

← Older revision	Revision as of 10:10, 13 July 2022
(No difference)

DMSadek at 06:38, 7 July 2022

2022-07-07T06:38:06Z

← Older revision		Revision as of 06:38, 7 July 2022
Line 33:		Line 33:
	=== General ===		=== General ===
	Quickly find known vulnerabilities for a certain application.		Quickly find known vulnerabilities for a certain application.
	Runs on windows and ~~Mac~~.		Runs on windows and Linux.

	=== Use ===		=== Use ===

DMSadek: Removed redirect to Web Scraper

2022-07-07T06:37:00Z

Removed redirect to Web Scraper

← Older revision		Revision as of 06:37, 7 July 2022
Line 1:		Line 1:
	~~#REDIRECT [[Web Scraper]]~~

	== Summary ==		== Summary ==
	Web Scraping is a way of automated data mining.		Web Scraping is a way of automated data mining.

DMSadek at 06:32, 7 July 2022

2022-07-07T06:32:27Z

← Older revision		Revision as of 06:32, 7 July 2022
Line 8:		Line 8:

	== Web crawler and Web scraper – not the same, but working together ==		== Web crawler and Web scraper – not the same, but working together ==
	These two get often confused and are sometimes even used as synonyms, but ~~looking at them~~ there is a different purpose and function to them.		These two get often confused and are sometimes even used as synonyms, but on closer inspection there is a different purpose and function to them.

	=== Web Crawler ===		=== Web Crawler ===

DMSadek at 06:31, 7 July 2022

2022-07-07T06:31:00Z

DMSadek at 06:29, 7 July 2022

2022-07-07T06:29:47Z

← Older revision		Revision as of 06:29, 7 July 2022
Line 4:		Line 4:
	Web Scraping is a way of automated data mining.		Web Scraping is a way of automated data mining.
	This means it is used for collecting various information in an automated way from open sources.		This means it is used for collecting various information in an automated way from open sources.
	The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data.		The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data. <br />
	There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.		There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.

DMSadek at 19:54, 6 July 2022

2022-07-06T19:54:52Z

← Older revision		Revision as of 19:54, 6 July 2022
Line 6:		Line 6:
	The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data.		The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data.
	There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.		There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.

			== Web crawler and Web scraper – not the same, but working together ==
			These two get often confused and are sometimes even used as synonyms, but looking at them there is a different purpose and function to them.

			=== Web Crawler ===
			Compared to the Web scraper you do not have a specified URL and you don’t know the domain the crawler will look at. Actually, that is the purpose of the crawler, you want to get the URLs, with which you might for example scrape later.
			So the output here is a list of URLs that might include what you were looking for.

			=== Web scraper ===

			The purpose of web crawling is about getting the desired data from specific websites. Here you usually know what data you want and the specific website you want it from, this is the most significant difference.
			Here the output is usually the data we specifically “ask” for.

			== How Can Web Scraping Enhance Cyber Security? ==
			Even though web scraping activities fundamentally do not aim to harm the website they are targeting, there are still some concerns that one should be aware of.
			Some scraper bots purposefully perform malicious attacks, but there are also those issues that aren’t on purpose.
			Some of the dangers are: extracting sensitive data, extract web apps, help attackers identify vulnerabilities, overwhelming the website with too many requests and more.
			== Pen testing ==

			While web crawler have enough concerning properties, it can also be useful in cyber security, it can for example be used for penetration testing.
			With the web crawler and web scraper it is possible to find possible vulnerabilities and remodel it in order to get rid of the vulnerability.


			== Our web scraper ==

			=== General ===
			Quickly find known vulnerabilities for a certain application.
			Runs on windows and Mac.

			=== Use ===

			==Sources==
			https://realpython.com/python-web-scraping-practical-introduction/#your-first-web-scraper
			https://www.cyberdb.co/how-web-scraping-can-enhance-cyber-security/
			https://thehackernews.com/2020/09/zenscrape-simple-web-scraping-solution.html

DMSadek at 19:51, 6 July 2022

2022-07-06T19:51:39Z

← Older revision		Revision as of 19:51, 6 July 2022
Line 1:		Line 1:
	#REDIRECT [[Web Scraper]]		#REDIRECT [[Web Scraper]]

	== ~~Level 2~~ ==~~Summary~~		== Summary ==
	Web Scraping is a way of automated data mining.		Web Scraping is a way of automated data mining.
	This means it is used for collecting various information in an automated way from open sources.		This means it is used for collecting various information in an automated way from open sources.
	The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data.		The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data.
	There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.		There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.

← Older revision		Revision as of 06:31, 7 July 2022
Line 3:		Line 3:
	== Summary ==		== Summary ==
	Web Scraping is a way of automated data mining.		Web Scraping is a way of automated data mining.
	This means it is used for collecting various information in an automated way from open sources.		This means it is used for collecting various information in an automated way from open sources.<br />
	The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data. <br />		The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data. <br />
	There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.		There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.
Line 11:		Line 11:

	=== Web Crawler ===		=== Web Crawler ===
	Compared to the Web scraper you do not have a specified URL and you don’t know the domain the crawler will look at. Actually, that is the purpose of the crawler, you want to get the URLs, with which you might for example scrape later.		Compared to the Web scraper you do not have a specified URL and you don’t know the domain the crawler will look at.<br />
			Actually, that is the purpose of the crawler, you want to get the URLs, with which you might for example scrape later.<br /><br />
	So the output here is a list of URLs that might include what you were looking for.		So the output here is a list of URLs that might include what you were looking for.

	=== Web scraper ===		=== Web scraper ===

	The purpose of web crawling is about getting the desired data from specific websites. Here you usually know what data you want and the specific website you want it from, this is the most significant difference.		The purpose of web crawling is about getting the desired data from specific websites. Here you usually know what data you want and the specific website you want it from, this is the most significant difference.<br /><br />
	Here the output is usually the data we specifically “ask” for.		Here the output is usually the data we specifically “ask” for.

	== How Can Web Scraping Enhance Cyber Security? ==		== How Can Web Scraping Enhance Cyber Security? ==
	Even though web scraping activities fundamentally do not aim to harm the website they are targeting, there are still some concerns that one should be aware of.		Even though web scraping activities fundamentally do not aim to harm the website they are targeting, there are still some concerns that one should be aware of.<br />
	Some scraper bots purposefully perform malicious attacks, but there are also those issues that aren’t on purpose.		Some scraper bots purposefully perform malicious attacks, but there are also those issues that aren’t on purpose.<br />
	Some of the dangers are: extracting sensitive data, extract web apps, help attackers identify vulnerabilities, overwhelming the website with too many requests and more.		Some of the dangers are: extracting sensitive data, extract web apps, help attackers identify vulnerabilities, overwhelming the website with too many requests and more.

	== Pen testing ==		== Pen testing ==

	While web crawler have enough concerning properties, it can also be useful in cyber security, it can for example be used for penetration testing.		While web crawler have enough concerning properties, it can also be useful in cyber security, it can for example be used for penetration testing.<br />
	With the web crawler and web scraper it is possible to find possible vulnerabilities and remodel it in order to get rid of the vulnerability.		With the web crawler and web scraper it is possible to find possible vulnerabilities and remodel it in order to get rid of the vulnerability.

Line 38:		Line 40:

	==Sources==		==Sources==
	https://realpython.com/python-web-scraping-practical-introduction/#your-first-web-scraper		https://realpython.com/python-web-scraping-practical-introduction/#your-first-web-scraper<br />
	https://www.cyberdb.co/how-web-scraping-can-enhance-cyber-security/		https://www.cyberdb.co/how-web-scraping-can-enhance-cyber-security/<br />
	https://thehackernews.com/2020/09/zenscrape-simple-web-scraping-solution.html		https://thehackernews.com/2020/09/zenscrape-simple-web-scraping-solution.html<br />