Top 16 open source intelligence tools (OSINT) to find anyone online
Top 16 open source intelligence tools
(OSINT) to find anyone online
What are the top 16, open source tools (OSINT) to find anyone online? What a great question! and a huge one.
We spend time talking to OSINT experts and private investigators to find out. Here we cover their good and bad points (not many bad points!), giving our review, link and advice.
This document will increase your knowledge and add 16 amazing investigation tools, leading you deeper into the dark and quite often secret cave, of online investigations.
As I’m sure you know, Open source intelligence (OSINT) is the application of utilising data that is publicly available, to achieve your aim.
Open source may give the impression of publicly available information only. OSINT could also utilise human intelligence (such as social engineering). But, there are many methods of gathering and sifting this intelligence, not all, but most, include using computer systems.
Why is OSINT so important?OSINT is making sense of the chaos that is online data (and sometimes offline). Anyone can gather information, with the right tools. True OSINT is making the links between the information to achieve a goal. OSINT tools provide solutions, simple. Solutions, to make your life better.
Using OSINT to discover public-facing assets
One of the more common aspects of starting an OSINT gather is attempting to pinpoint public facing assets, such as websites, company address, personal address. We’re not really interested in system or computer program vulnerabilities, that’s more for the penetration testing world.
However, a little bit of penetration testing knowledge can glean some information. For e.g., remotely connect to a server system owned by your target, you can get some intel. Prior entering a username or password, you’re presented with the operating system and contact details of the administrator. Sometimes, this can be useful in social engineering situations.
Using OSINT to discover relevant information outside the organization
A secondary function of OSINT tools these days is to find ‘extra’ relevant information outside of your direct query that you have looked for. Such information could be an aquired company, secondary company, or close relationships or contacts to the target in quesiton. This can identify further potential physical address locations of the target. Other interests such as social network accounts and social life pictures, can provide a picture. That picture may show the local social habitat and the target. You may now be aware of their local stomping grounds online.
Collate discovered information into actionable form
Finally, once the information has been searched, collected, including the connected relevant information, someone needs to group this all together. They should be linking all the points together and apply some logical analysis (an investigative mindset required!).
Running OSINT scans can return large volumes of information, you need to be well versed at focusing your time on the information that matters, with a keen eye for detail. Piecing all that information together to focus on the most useful aspects can be very effective in painting the larger picture. Tools try to do this, but really, this part needs a good human brain and some coffee.
Top OSINT tools used bv the Experts
The first job to do? choose the right tool(s!). You dont nessisarly need to use any tools, but given the vast locations where information can be, having a tool, helps. A tool that checks thousands of locations within a few seconds can be very time-efficient.
If you’re trying to discover information on your own identity or business, preventing future online attackers, you want something quick and repeatable. Doing this quickly could mean the difference between your employees or family from recieving phishing attacks.
So, here you have it. The top tools we think will help you to conduct a thourough, and effective, OSINT analysis of either your target, or your own assets. We list them here, then go into detail of how each one can make your investigative career, more effective (in no particular order):
- Maltego
- UserSearch.org
- Mitaka
- SpiderFoot
- Spyse
- BuiltWith
- Intelligence X
- DarkSearch.io
- Grep.app
- Recon-ng
- theHarvester
- Shodan
- Metagoofil
- Searchcode
- SpiderFoot
- Babel X
Maltego : making complex OSINT easy, with great maps and transforms
Maltego focus’s on finding particular relationships between assets, people, companies and website domains (and it does this very well). Most major OSINT platforms provide Maltego with an API, these days. They let the tool hook in and present the data in Maltego. However, it can be known to take a large amount of time to time to plot all the raw intelligence.
The graphs in the tool do an amazing job at presenting the information in easy-to-read charts. The latest we heard, it can show up to 10,000 data points.
The tool works by automating the search of your input, against various public information sources. It attempts to provide a 1 click to source solution, and we think it does this very well. It manages these data sources by something they call a ‘transform’, and it comes with quite a few inbuilt features. Some of these are DNS records, social network searching, WHOIS checks and a few others. You can also add custom transforms if find a compatible API that you want to hook it up.
There is a free version with limited features and a charged instance from $1,999. They also provide server installation solutions for large-scale solutions, which starts at $40,000 and includes intensive training.
UserSearch : Finding people by usernames and Email
UserSearch, is a vast network of search engines that crawl the web to find an exact match on a username or email address. It scans across hundreds and hundreds of websites. It can locate a particular user profile on Forums, Social networks, dating websites, message boards, crypto websites, gambling websites. Its by far one of the most comprehensive and accurate search engines online for usernames and an email address.
Its been running for over 15 years, so the developers have a pretty strong position for OSINT of usernames or email addresses. The developers are a team of open source experts and software developers.
The site is totally free to use and it can check up to 800 sites within about 30 seconds, which is pretty amazing. Its quite literally the Google, for usernames and email addresses.
They also recently released a premium service that is in Alpha version. It reports to find even more information and cover more websites, such as dating sites, within the premium version. They offer the service for $6.99 per month which you can cancel at any time.
Mitaka : finding IPs, MD5s, ASNs and bitcoin address’s
This is available as a Chrome Extension and also for Firefox. It allows you to search over a dozen major search engines for domains, URLs, IP address’s, MD5 hashes, ASNs, Bitcoin address’s.
Its a very handy set of tools in your browser, and for those who may prefer a more focused, limited set of tools.
Spiderfoot : vast array of open source intelligence resources
This is a free recon tool that can pull data from multiple different sources to glean information on various online assets. It searches for IP address, CIDR ranges, domains, subdomains, ASNs, email addresses and phone numbers. Its on Github and comes in both a command line version and a embedded web-server version. It contains over 200 modules, making it perfect for red and blue teams as part of your initial process.
Spyse : the OSINT goto tool for domains
This tool is described online as ‘the most complete internet asset registry‘ online. Its main focus seems to be leaning towards cyber security work. The tool is used by many major OSINT tools, providing the back end data. It collects publicly available data on websites, their WHOIS information (such as owners, associated servers and IoT linked devices).
The data can then be reviewed by their own engine to identify potential security risks and connections between various entities. It does have a free plan, although developers who hope to build applications from the data being provided by the Spyse API, will need to pay a subscription.
BuiltWith : OSINT to find what sites are built with!
This tool is perfectly named. Using this tool, you can determine what technology stacks are used on various websites and platforms. It can, as an example, detect whether a website is built using WordPress, Joomla, or any other CMS-like platform. It will also generate a great service list of the plug-ins that the website is currently using, its frameworks and even its sever information that’s publicly available. Its anther great tool for red and blue teams in initial server recon as part of a security audit.
As an additional bonus, you can link this tool in with a security scanner, such as WPScan, and the WordPress Vulnerability Database API. This can then quickly spot common vulnerabilities within the WordPress modules being used.
Intelligence X : a database of literally everything OSINT!
Intelligence x is one of the best archival services and search services that we have to access online. Not only does it archive historic versions of websites online but it also includes leaked data that is typically removed quite quickly, these days. Although this may sound a little similar to what Internet Archive’s service offers, Intelligence X offers some pretty cool differences. No matter how controversial the data leak, Intelligence X seems to not worry too much, providing it public for the masses, forever.
Previously, its archived information such as the email servers during the leak of the Hillary Clinton and Donald Trump situations.
Grep.app
Ever thought that if you could search across half a million git repos on a couple of clicks, would be handy? sure you may have tried using the search bar on github. But, Grep.app does the job far quicker, and more effectivly.
Recon-ng : great for python scripters
Do you a little bit of Python coding? If so, you have an early Xmas present when it comes to recon-ng. This is a very powerful python tool, which is written in python. It allows you to interface very nicely into MetaSploit Framework, which should make matters easier for you to learn its uses. It also has a help area which guides you on best uses of the tool, so python developers should be able to integrate it pretty quickly.
The tool itself automates one of the most time-consuming aspects of OSINT actions. It allows you to automate more of the time consuming repetitive actions, allowing you to focus on the actual OSINT investigation.
Its designed so even the newest python developer should be able to utilise the tool to search public information and return some decent results. It works in modules, with a lot of inbuilt functionality designed for easy use. Common tasks such as normalising the data outputs, linking into databases and making URL requests to websites and using / managing API keys to gather data direct from API interfaces, are fast and easy to do with Python. Instead of needing to program Recon-ng, it allows you to simply choose what module you want to use and give it the destination.
The tool is free, open source and it includes a huge wiki that comprehensively includes information on how to get started and used the tool, using best practice for OSINT.
theHarvester : OSINT for networks
One of the easier tools to use out of the box that we’ve come across, is the popular tool theHarvester. Its build from the bottom up to gather public information that exists on an organisation, but outside the organisations own network. It can identify interesting public information on an organisation computer network, from looking outside the fence. Using a multitude of inbuilt tools to do this, making it a very effective reconnaissance tool prior starting a penetration test or a similar activity.
The tool gathers the information from major data providers such as Bing and Google, but also using less known areas of the web and meta engines.
To do this it hooks into Netcraft for data mining and the Alien vault threat exchange, allowing it to quickly identify known vulnerabilities. In basic, it can port scan, gather emails, names, sub domains, IPs and URLs on an organisation, perfectly positioning you for the next steps of you’re exercise.
Shodan : the Google for IoT devices
Shodan is an amazing, dedicated search engine that’s used to find intelligence on IoT devices. These kind of devices are not normally searchable, so you’ll be surprised at what you can find out.
It can detect open ports, vulnerabilities on targeted systems and scan devices that are not typically supported by standard port scanners.
Other OSINT tools like theHarvester, actually use Shodan as a data source for detecting vulnerabilities on IoT devices. One of the greatest advantages of Shodan is its to purely monitor so many hundreds of thousands of IoT devices and the information it contains on these devices, making it publicly available. If you don’t use this tool at the start of an OSINT, your missing a huge amount of data. Devices such as cameras, building sensors, security devices, xboxes, security cameras, household fridges! the list is endless.
This tool is a charged service, but well worth the cost. It costs $59 per month at basic. But to test it out, you can register a freelancer licence and scan up to 5120 IPs per month for free.
Metagoofil : who made that document?
This is a freely available tool on Github. Its designed and optimised to pull out information and meta data on public documents, such as PDF, Doc, docx, xls and other common document formats. Its literally an online, document investigation tool.
The kind of information you can pull from this tool is very interesting, and impressive. It can return artefacts such as the username of who created the document, as well as their real name and sometimes their address (if the computer used to create the document contains the registered system details containing this information). Also artefacts such as server name, network share resources and directory information of the network shares.
It It goes without saying, this kind of data leakage from an organisation is very useful for anyone who works in the OSINT field, or perhaps needs to conduct some social engineering activities. A very valuable and free resource for any OSINT investigator.
searchcode : the Google of code!
Searchcode is essentially the Google equivalent of software code. Software developers quite often leave sensitive information within the source code of computer programs, on the assumption no-one will ever look. You can come across emails, usernames and even passwords inside source code. This tool searches the source code, quickly finding useful forgotten pieces.
You dont need to be a devleoper or coder to understand the results of this search engine, and most of the time the results You don’t need to be a developer or coder to understand the results of this search engine, and most of the time the results are self explanatory. Another great resource for OSINT investigators where they know their targets or organisations may have released software.
Babel X : Open source intelligence in Chinese, Russian, Tamil…
Do you ever need to conduct an OSINT investigation on a target and they may speak another language but you don’t? this could be a problem. From our research, about a quarter of the internet users worldwide speak English as their first language, although some sources say its 55%. What about all that useful information that we cant see if were not fluent in Spanish, Chinese, Russian, or Tamil?
This tool is the solution to that problem. It lets you search the public web, such as blogs, social media and common message boards, plus the dark web and deep web. It can currently search against data sources of more than 200 languages.
This is a hugely useful resource and OSINT investigators should definatly try this out.
OSINT Framework Resource : The Best of the best
All in all, what is the best resource online?
The OSINT Framework resource. Why? its an amazing resource to seeing far more online resources for OSINT. Its very, very, popular. OSINT Framework website has a huge array of OSINT resources for all walks of life.
Comments
Post a Comment