View source for CVE Webscraper

#REDIRECT [[Web Scraper]]

== Summary ==
Web Scraping is a way of automated data mining.
This means it is used for collecting various information in an automated way from open sources.
The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data.
There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.

== Web crawler and Web scraper – not the same, but working together ==
These two get often confused and are sometimes even used as synonyms, but looking at them there is a different purpose and function to them.

=== Web Crawler ===
Compared to the Web scraper you do not have a specified URL and you don’t know the domain the crawler will look at. Actually, that is the purpose of the crawler, you want to get the URLs, with which you might for example scrape later.
So the output here is a list of URLs that might include what you were looking for.

=== Web scraper ===

The purpose of web crawling is about getting the desired data from specific websites. Here you usually know what data you want and the specific website you want it from, this is the most significant difference.
Here the output is usually the data we specifically “ask” for.

== How Can Web Scraping Enhance Cyber Security? ==
Even though web scraping activities fundamentally do not aim to harm the website they are targeting, there are still some concerns that one should be aware of.
Some scraper bots purposefully perform malicious attacks, but there are also those issues that aren’t on purpose.
Some of the dangers are: extracting sensitive data, extract web apps, help attackers identify vulnerabilities, overwhelming the website with too many requests and more.
== Pen testing ==

While web crawler have enough concerning properties, it can also be useful in cyber security, it can for example be used for penetration testing.
With the web crawler and web scraper it is possible to find possible vulnerabilities and remodel it in order to get rid of the vulnerability.


== Our web scraper ==

=== General ===
Quickly find known vulnerabilities for a certain application.
Runs on windows and Mac.

=== Use ===

==Sources==
https://realpython.com/python-web-scraping-practical-introduction/#your-first-web-scraper
https://www.cyberdb.co/how-web-scraping-can-enhance-cyber-security/
https://thehackernews.com/2020/09/zenscrape-simple-web-scraping-solution.html