OSINT Tools to Use
OSINT tools: An expanding list
Maltego : making complex OSINT easy, with great maps and transforms
Maltego focus’s on finding particular relationships between assets, people, companies and website domains (and it does this very well). Most major OSINT platforms provide Maltego with an API, these days. They let the tool hook in and present the data in Maltego. However, it can be known to take a large amount of time to time to plot all the raw intelligence.
The graphs in the tool do an amazing job at presenting the information in easy-to-read charts. The latest we heard, it can show up to 10,000 data points.
The tool works by automating the search of your input, against various public information sources. It attempts to provide a 1 click to source solution, and we think it does this very well. It manages these data sources by something they call a ‘transform’, and it comes with quite a few inbuilt features. Some of these are DNS records, social network searching, WHOIS checks and a few others. You can also add custom transforms if find a compatible API that you want to hook it up.
There is a free version with limited features and a charged instance from $1,999. They also provide server installation solutions for large-scale solutions, which starts at $40,000 and includes intensive training.
UserSearch : Finding people by usernames and Email
UserSearch, is a vast network of search engines that crawl the web to find an exact match on a username or email address. It scans across hundreds and hundreds of websites. It can locate a particular user profile on Forums, Social networks, dating websites, message boards, crypto websites, gambling websites. Its by far one of the most comprehensive and accurate search engines online for usernames and an email address.
Its been running for over 15 years, so the developers have a pretty strong position for OSINT of usernames or email addresses. The developers are a team of open source experts and software developers.
The site is totally free to use and it can check up to 800 sites within about 30 seconds, which is pretty amazing. Its quite literally the Google, for usernames and email addresses.
They also recently released a premium service that is in Alpha version. It reports to find even more information and cover more websites, such as dating sites, within the premium version. They offer the service for $6.99 per month which you can cancel at any time.
Mitaka : finding IPs, MD5s, ASNs and bitcoin address’s
This is available as a Chrome Extension and also for Firefox. It allows you to search over a dozen major search engines for domains, URLs, IP address’s, MD5 hashes, ASNs, Bitcoin address’s.
Its a very handy set of tools in your browser, and for those who may prefer a more focused, limited set of tools.
Spiderfoot : vast array of open source intelligence resources
This is a free recon tool that can pull data from multiple different sources to glean information on various online assets. It searches for IP address, CIDR ranges, domains, subdomains, ASNs, email addresses and phone numbers. Its on Github and comes in both a command line version and a embedded web-server version. It contains over 200 modules, making it perfect for red and blue teams as part of your initial process.
Spyse : the OSINT goto tool for domains
This tool is described online as ‘the most complete internet asset registry‘ online. Its main focus seems to be leaning towards cyber security work. The tool is used by many major OSINT tools, providing the back end data. It collects publicly available data on websites, their WHOIS information (such as owners, associated servers and IoT linked devices).
The data can then be reviewed by their own engine to identify potential security risks and connections between various entities. It does have a free plan, although developers who hope to build applications from the data being provided by the Spyse API, will need to pay a subscription.
BuiltWith : OSINT to find what sites are built with!
This tool is perfectly named. Using this tool, you can determine what technology stacks are used on various websites and platforms. It can, as an example, detect whether a website is built using WordPress, Joomla, or any other CMS-like platform. It will also generate a great service list of the plug-ins that the website is currently using, its frameworks and even its sever information that’s publicly available. Its anther great tool for red and blue teams in initial server recon as part of a security audit.
As an additional bonus, you can link this tool in with a security scanner, such as WPScan, and the WordPress Vulnerability Database API. This can then quickly spot common vulnerabilities within the WordPress modules being used.
Intelligence X : a database of literally everything OSINT!
Intelligence x is one of the best archival services and search services that we have to access online. Not only does it archive historic versions of websites online but it also includes leaked data that is typically removed quite quickly, these days. Although this may sound a little similar to what Internet Archive’s service offers, Intelligence X offers some pretty cool differences. No matter how controversial the data leak, Intelligence X seems to not worry too much, providing it public for the masses, forever.
Previously, its archived information such as the email servers during the leak of the Hillary Clinton and Donald Trump situations.
Ever thought that if you could search across half a million git repos on a couple of clicks, would be handy? sure you may have tried using the search bar on github. But, Grep.app does the job far quicker, and more effectivly.
Recon-ng : great for python scripters
Do you a little bit of Python coding? If so, you have an early Xmas present when it comes to recon-ng. This is a very powerful python tool, which is written in python. It allows you to interface very nicely into MetaSploit Framework, which should make matters easier for you to learn its uses. It also has a help area which guides you on best uses of the tool, so python developers should be able to integrate it pretty quickly.
The tool itself automates one of the most time-consuming aspects of OSINT actions. It allows you to automate more of the time consuming repetitive actions, allowing you to focus on the actual OSINT investigation.
Its designed so even the newest python developer should be able to utilise the tool to search public information and return some decent results. It works in modules, with a lot of inbuilt functionality designed for easy use. Common tasks such as normalising the data outputs, linking into databases and making URL requests to websites and using / managing API keys to gather data direct from API interfaces, are fast and easy to do with Python. Instead of needing to program Recon-ng, it allows you to simply choose what module you want to use and give it the destination.
The tool is free, open source and it includes a huge wiki that comprehensively includes information on how to get started and used the tool, using best practice for OSINT.
theHarvester : OSINT for networks
One of the easier tools to use out of the box that we’ve come across, is the popular tool theHarvester. Its build from the bottom up to gather public information that exists on an organisation, but outside the organisations own network. It can identify interesting public information on an organisation computer network, from looking outside the fence. Using a multitude of inbuilt tools to do this, making it a very effective reconnaissance tool prior starting a penetration test or a similar activity.
The tool gathers the information from major data providers such as Bing and Google, but also using less known areas of the web and meta engines.
To do this it hooks into Netcraft for data mining and the Alien vault threat exchange, allowing it to quickly identify known vulnerabilities. In basic, it can port scan, gather emails, names, sub domains, IPs and URLs on an organisation, perfectly positioning you for the next steps of you’re exercise.
Shodan : the Google for IoT devices
Shodan is an amazing, dedicated search engine that’s used to find intelligence on IoT devices. These kind of devices are not normally searchable, so you’ll be surprised at what you can find out.
It can detect open ports, vulnerabilities on targeted systems and scan devices that are not typically supported by standard port scanners.
Other OSINT tools like theHarvester, actually use Shodan as a data source for detecting vulnerabilities on IoT devices. One of the greatest advantages of Shodan is its to purely monitor so many hundreds of thousands of IoT devices and the information it contains on these devices, making it publicly available. If you don’t use this tool at the start of an OSINT, your missing a huge amount of data. Devices such as cameras, building sensors, security devices, xboxes, security cameras, household fridges! the list is endless.
This tool is a charged service, but well worth the cost. It costs $59 per month at basic. But to test it out, you can register a freelancer licence and scan up to 5120 IPs per month for free.
Metagoofil : who made that document?
This is a freely available tool on Github. Its designed and optimised to pull out information and meta data on public documents, such as PDF, Doc, docx, xls and other common document formats. Its literally an online, document investigation tool.
The kind of information you can pull from this tool is very interesting, and impressive. It can return artefacts such as the username of who created the document, as well as their real name and sometimes their address (if the computer used to create the document contains the registered system details containing this information). Also artefacts such as server name, network share resources and directory information of the network shares.
It It goes without saying, this kind of data leakage from an organisation is very useful for anyone who works in the OSINT field, or perhaps needs to conduct some social engineering activities. A very valuable and free resource for any OSINT investigator.
searchcode : the Google of code!
Searchcode is essentially the Google equivalent of software code. Software developers quite often leave sensitive information within the source code of computer programs, on the assumption no-one will ever look. You can come across emails, usernames and even passwords inside source code. This tool searches the source code, quickly finding useful forgotten pieces.
You dont need to be a devleoper or coder to understand the results of this search engine, and most of the time the results You don’t need to be a developer or coder to understand the results of this search engine, and most of the time the results are self explanatory. Another great resource for OSINT investigators where they know their targets or organisations may have released software.
Tineye is a tool used for searching images on the web. For instance, it uses machine language, neural networks, and pattern recognition to get results on millions of images online. With the tool you can discover if an image has been uploaded anywhere online, and the exact location where it was uploaded. The tool features watermark identification, image matching, Tineye alert system, mobile engine, colour search API, and signature identification.
CheckUserNames is an online OSINT tool that can help you to find usernames across over 170 social networks. This is especially useful if you are running an investigation to determine the usage of the same username on different social networks.
It can be also used to check for brand company names, not only individuals.
HaveIbeenPwned can help you to check if your account has been compromised in the past. This site was developed by Troy Hunt, one of the most respected IT security professionals of this market, and it’s been serving accurate reports since years.
If you suspect your account has been compromised, or want to verify for 3rd party compromises on external accounts, this is the perfect tool. It can track down web compromise from many sources like Gmail, Hotmail, Yahoo accounts, as well as LastFM, Kickstarter, Wordpress.com, Linkedin and many other popular websites.
Once you introduce your email address, the results will be displayed, showing something like:
BeenVerified is another similar tool that is used when you need to search people on public internet records. It can be pretty useful to get more valuable information about any person in the world when you are conducting an IT security investigation and a target is an unknown person.
After done, the results page will be displayed with all the people that match the person’s name, along with their details, geographic location, phone number, etc. Once found, you can build your own reports.
The amazing thing about BeenVerified it’s that it also includes information about criminal records and official government information as well.
BeenVerified background reports may include information from multiple databases, bankruptcy records, career history, social media profiles and even online photos.
Censys is a wonderful search engine used to get the latest and most accurate information about any device connected to the internet, it can be servers or domain names.
You will be able to find full geographic and technical details about 80 and 443 ports running on any server, as well as HTTP/S body content & GET response of the target website, Chrome TLS Handshake, full SSL Certificate Chain information, and WHOIS information.
While investigating people or companies, a lot of IT security newbies forget the importance of using traditional search engines for recon and intel gathering.
In this case, Google Dorks can be your best friend. They have been there since 2002 and can help you a lot in your intel reconnaissance.
Google Dorks are simply ways to query Google against certain information that may be useful for your security investigation.
Search engines index a lot of information about almost anything on the internet, including individual, companies, and their data.
Some popular operators used to perform Google Dorking:
- Filetype: you can use this dork to find any kind of filetypes.
- Ext: can help you to find files with specific extensions (eg. .txt, .log, etc).
- Intext: can perform queries helps to search for specific text inside any page.
- Intitle: it will search for any specific words inside the page title.
- Inurl: will look out for mentioned words inside the URL of any website.
Log files aren’t supposed to be indexed by search engines, however, they do, and you can get valuable information from these Google Dorks, as you see below:
Jigsaw is used to gather information about any company employees. This tool works perfectly for companies like Google, Linkedin, or Microsoft, where we can just pick up one of their domain names (like google.com), and then gather all their employee’s emails on the different company departments.
The only drawback is that these queries are launched against Jigsaw database located at jigsaw.com, so, we depend entirely on what information they allow us to explore inside their database. You will be able to find information about big companies, but if you are exploring a not so famous startup then you may be out of luck.
Creepy is a geo-location OSINT tool for infosec professionals. It offers the ability to get full geolocation data from any individuals by querying social networking platforms like Twitter, Flickr, Facebook, etc.
If anyone uploads an image to any of these social networks with geolocation feature activated, then you will be able to see a full active mal where this person has been.
You will be able to filter based on exact locations, or even by date. After that, you can export the results in CSV or KML format.
Nmap is one of the most popular and widely used security auditing tools, its name means “Network Mapper”. Is a free and open source utility utilized for security auditing and network exploration across local and remote hosts.
Some of the main features include:
- Host detection: Nmap has the ability to identify hosts inside any network that have certain ports open, or that can send a response to ICMP and TCP packets.
- IP and DNS information detection: including device type, Mac addresses and even reverse DNS names.
- Port detection: Nmap can detect any port open on the target network, and let you know the possible running services on it.
- OS detection: get full OS version detection and hardware specifications of any host connected.
- Version detection: Nmap is also able to get application name and version number.
WebShag is a great server auditing tool used to scan HTTP and HTTPS protocols. Same as other tools, it’s part of Kali Linux and can help you a lot in your IT security research & penetration testing.
You will be able to launch a simple scan, or use advanced methods like through a proxy, or over HTTP authentication.
Written in Python, it can be one of your best allies while auditing systems.
Main features include:
- Port Scan
- URL scanning
- File fuzzing
- Website crawling
In order to avoid getting blocked by remote server security systems, it uses an intelligent IDS evasion system by launching random requests per HTTP proxy server, so you can keep auditing the server without being banned.
OpenVAS (Open Vulnerability Assessment System) is a security framework that includes particular services and tools for infosec professionals.
This is an open source vulnerability scanner & security manager that was built after the famous Nessus switched from open source to private source. Then, the original developers of the Nessus vulnerability scanner decided to fork the original project and create OpenVAS.
While it is a little bit more difficult to setup than the old Nessus, it’s quite effective while working with it to analyze the security of remote hosts.
The main tool included in OpenVAS is OpenVAS Scanner, a highly efficient agent that executes all the network vulnerability tests over the target machine.
On the other hand, another main component is called OpenVAS Manager, which is basically vulnerability management solution that allows you to store scanned data into an SQLite database, so then you can search, filter and order the scan results in a fancy and easy way.
Fierce is an IP and DNS recon tool written in PERL, famous for helping IT sec professionals to find target IPs associated with domain names.
It was written originally by RSnake along with other members of the old http://ha.ckers.org/. It’s used mostly targetting local and remote corporate networks.
Once you have defined your target network, it will launch several scans against the selected domains and then it will try to find misconfigured networks and vulnerable points that can later leak private and valuable data.
The results will be ready within a few minutes, a little bit more than when you perform any other scan with similar tools like Nessus, Nikto, Unicornscan, etc.
Unicornscan is one of the top intel gathering tools for security research. It has also a built-in correlation engine that aims to be efficient, flexible and scalable at the same time.
Main features include:
- Full TCP/IP device/network scan.
- Asynchronous stateless TCP scanning (including all TCP Flags variations).
- Asynchronous TCP banner detection.
- UDP Protocol scanning.
- A/P OS identification.
- Application and component detection.
- Support for SQL Relational Output
FOCA (Fingerprinting Organizations with Collected Archives) is a tool written by ElevenPaths that can be used to scan, analyze, extract and classify information from remote web servers and their hidden information.
Foca has the ability to analyze and collect valuable data from MS Office suite, OpenOffice, PDF, as well as Adobe InDesign and SVG and GIF files. This security tool also works actively with Google, Bing and DuckDuckGo search engines to collect additional data from those files. Once you have the full file list, it starts extracting information to attempt to identify more valuable data from the files.
In the cybersecurity world, we researchers are used to popular IoT search engines such as Shodan or Censys. For a while, however, a powerful new IoT search engine has been rapidly gaining followers. We’re talking about ZoomEye.
ZoomEye is a Chinese IoT OSINT search engine that allows users to grab public data from exposed devices and web services. In order to build its database it uses Wmap and Xmap, and then runs extensive fingerprinting against all the information found, ultimately presenting it to users in a filtered and curated way for easy visualization.
What information can you find with ZoomEye?
- IPs interacting with networks and hosts
- Open ports on remote servers
- Total number of hosted websites
- Total number of devices found
- Interactive map of users hitting different devices
- Vulnerabilities report
And much more. The public version offers access to a lot of data — but if you want to see what it can really do, we suggest you sign up for a free account. That way you’ll get to test the real power of this OSINT tool.
Wappalyzer is a highly useful service that allows security researchers to quickly identify technologies on websites. With it, you can find a complete list of details for any technology stack running on any website. It also allows you to build lists of websites that use certain technologies, letting you add phone numbers and email addresses as well.
Their free plan includes instant results and up to 50 free monthly lookups. It’s perfect for tracking website technologies, discovering old/vulnerable software, finding organic data about your competitors, and last but not least, can be quickly triggered from the web browser with their Chrome/Firefox extensions.
If that isn’t enough, they also offer a handy API to automate technology lookups, and you can even set up website alerts to monitor your competition.
This infosec tool is frequently overlooked, but it has great potential in boosting your infosec discovery and analysis processes. IVRE is an open source tool that’s built on a base of popular projects like Nmap, Masscan, ZDNS, and ZGrab2.
Its framework uses these popular tools to gather network intelligence on any host, then uses a MongoDB database to store the data.
Its web-based interface makes it easy for both beginning and advanced infosec users to perform the following actions:
- Passive reconnaissance by flow analysis (from Zeek, Argus or nfdump)
- Active reconnaissance by using Zmap and Nmap
- Fingerprinting analysis
- Import data from other 3rd party infosec apps, such as Masscan/Nmap
IVRE can be installed by fetching the source from their official Github repo, or from 3rd-party repositories such as Kali Linux repo.
While a lot of OSINT tools focus on data found on public files such as PDF, .DOC, HTML, .SQL, etc., there are other tools that are specifically designed to extract critical Open Source Intelligence data from image, video and audio files.
Exiftool reads, writes and extracts metadata from the following types of files:
- And many others
It also supports native files from a wide range of cameras, such as: Canon, Casio, FujiFilm, Kodak, Sony, and many others. It’s also conveniently available on multiple platforms including Linux, Windows and MacOS.
Happy Hacking !!!