Week in OSINT #2022-09
Your weekly OSINT news is back with a collection of tools on Twitter, safe communications, media and spies...
For everyone into OSINT the last few weeks have been fascinating and gruesome at the same time. With all the news about the war in Ukraine and all the disinformation, I was stuck with loads of work. Besides that I was also ill for a few days, so I had to skip an episode for once. But I'm back, and tried to pick a few very interesting topics for this week. We'll look at Twitter, but also follow the live news, and I'm going to end with a small tutorial like item:
- Tweepdiff
- Digital Communication Protocols
- VidGrid
- Investigating Disinformation
- The Spy Collection
- Regular Expressions
Tool: Tweepdiff
I got a tip from @IntelLana about a new online tool by Brian Deterling called Tweepdiff. On it everyone can look for followers or accounts that are followed by two or more Twitter accounts. The handy thing is, that no login is required, simply fill in the account names and off you go.
Link: https://tweepdiff.com/
Tip: Digital Communication Protocols
Twitter user @officer_cia has shared a Google Sheet with technical information on different communication protocols and their respective apps. It lists the encryption techniques, privacy status, compatibility, features and more. It has links to technical information supporting certain claims. This is an interesting list, especially for anyone who has the need for secure communication. And while you're at it, also check out the GitHub repo of @officer_cia on OpSec, because that's worth looking at too!
Sheet: https://docs.google.com/spreadsheets/...
OpSec repo on GitHub https://github.com/OffcierCia/...
Site: VidGrid
Loránd Bodó shared a very interesting online tool called VidGrid by Matt Taylor. It enables you to view multiple news streams in your browser. Sounds for one stream can be enabled by clicking the desired video. A simple yet very useful tool when following the latest news.
Link: https://vidgrid.tk.gg/
Tip: Investigating Disinformation
Marc Owen Jones noticed some peculiar tweets and hashtags in the last weeks, and decided to dive into the world of Twitter bots. He gathered thousands of messages with NodeXL and looked into the way people on Twitter were interacting with these messages. He covers different topics, like engagement techniques and the creation date of used accounts. A very informative thread, thank you for sharing this with the world!
Thread 1/ This is a thread on pro-Russian propaganda & #disinformation. I analysed the hashtags "i stand with Putin" & "i stand with Russia'. I analysed around 20,000 Twitter interactions involving 9600 unique accounts
— Marc Owen Jones (@marcowenjones) March 3, 2022
Bots ✔️
Engagement Farming ✔️#UkraineRussianWar #Ukraine pic.twitter.com/I8XBwPlc7b
Thread: https://twitter.com/marcowenjones/status/1499312091727020032
Tip: The Spy Collection
Somehow the Twitter account @SpyCollection1 popped up in my timeline. For quite some time this person has a YouTube channel, talking about spy related gadgets and stories. But there's a Medium blog now too, with a weekly newsletter covering news from the world of espionage, cryptography and intelligence. This is an awesome source for curated stories, thank you for collecting all this!
Medium: https://medium.com/@thespycollection
YouTube: https://www.youtube.com/c/SpyCollectionChannel/
Tip: Regular Expressions
I once mentioned a browser plugin for Regular Expressions, in episode 2021-11, and when I saw a little tip last week I wanted to include it. But this time, I'll take the time to explain a bit more about regular expressions, and how it can be used.
The tip was shared online by multiple people, but it all started with @spiderfoot sharing a regular RegEx to dump email addresses straight into your console or command line:
curl -s [URL] | grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b"
Even though this does work for most email addresses, we should at least make some adjustments. Without going into the workings of regular expressions, this query doesn't include top level domains with more than 6 characters, have a dash in the domain name (like foreign TLDs that are encoded and start with xn--), or some other characters that are officially allowed in the name part of an email address. So we end up with the following rule that captures the most common email addresses out there. For now I am going to skip the whole list of special characters that are allowed, to keep it somewhat readable:
curl -s [URL] | grep -E -o "\b[a-zA-Z0-9._%$#=+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-]+\b"
For some of my readers, RegEx is something completely new, so for the rest of this Week in OSINT I'll take you into the basics of basic regular expressions. This query language is extremely powerful and can be really useful when dealing with data. Whether you need to extract email addresses from documents, find cryptocurrency addresses inside a webpage, or want to retrieve all URLs within a folder, it's all possible.
The Basics
When you write a regular expression, it's important to remember that it'll be going over text or data from left to right, top to bottom, just like how you are reading this text. And while it scans the stream of data, it constantly checks characters whether they match a set of rules. An expression can be built with several patterns in a row, to match different parts of a specific string, in the same order that you've created the rule. Do bear in mind that everything is case-sensitive, unless you specifically overrule this.
Syntax and Operations
I can list a whole range of different operators and patterns, and some things are easier to do than I'll be writing here, but to explain some basics I'll show the workings of some very basic regular expressions. Let's start with matching one of two words, or part of words. If you want to find any of the words red, blue or yellow, you simply list them and divide the options with a vertical bar or so called 'pipe':
red|blue|yellow
When you want to look for bits of data that only consists of numbers, you can create a collection of characters ranging from zero to nine. For that you use the square brackets to indicate it's a list of possible characters. The + sign behind it means: The matching character from the collection, should appear at least once, but can be repeated infinitely:
[0-9]+
The same goes for alphabetical characters. It's possible to search for every part of text that only has uppercase or lowercase letters (so excluding spaces, commas or such) like this:
[a-zA-Z]+
And you can change how far the alphabet goes, so it's possible to create a range from A to F, like we have in hexadecimal numbers, next to the usual numbers. To make sure we capture both :
[a-fA-F0-9]+
If you are only interested in hexadecimal numbers that have a specific length, you can follow a specific rule with some curly brackets. Include a single number, and the length has to be precise. If you include two numbers, you can specify the minimum and maximum length it should have. For instance, this will return all lowercase hexadecimal numbers that are between 2 and 6 characters in length:
[a-f0-9]{2,6}
Learning RegEx
This is just a small snippet of what's possible, and to show you that the basics of regular expressions, or RegEx, are relatively easy to grasp. When one creates extremely long queries, it might be difficult to understand what the exact working of it is, but there are tools that can help with this. An online tool that can help you write and test a query is RegEx101.com. While writing the query, or when you found a query and parse it in the editor, it shows you every step of the process and explains what everything is. Another and similar tool is RegExr.com, that offers similar features and offers a cheat sheet, and even has the ability to save your own queries online.
But if you are completely new to this, then these tools can maybe only confuse you. So why not start with an interactive online class, working your way through simple excercises over at RegExOne.com. Enjoy!
Have a good week and have a good search!
