Massive OSINT automation

Automation, discovery and aggregation of threat intelligence at scale to 10X analyst productivity.

We started with a request from management for a quick-to-generate automated report consisting of open-source intelligence based on the assets provided by out clients.

Our analysts and sales team used the report generation product so heavily that the demands quickly out grew the original use case and we needed to redesign the system to handle the success.

I lead the redesign effort. The crucial concept that made the new system work was forgetting about the idea of knowing when a report was completed. Instead, we built a system to continuously monitor client assets, giving users the ability to take a snapshot at any point in time.

We built the system with a plugin architecture, allowing us to easily add or remove OSINT sources to keep up with the needs of our analysts.

We built asset discovery into the system, so given just a domain or a company name to start with, the collection system would quicly discover dozens or hundreds of other assets and add them to the client’s monitoring profile.

We designed output around a vision of user-driven customizable reports. Analyst were able to specify the exact data they needed and the system would provide a report ready to hand to clients, needing only a manual review to provide expert validation. This enabled analysts to skip routine tasks and skip ahead to more complex analysis that needs a human touch.

The architecture of this system and the cloud-deployment allowed it to adapt to changing user requirements while scaling to handle large throughput for Fortune 500 clients and government organizations.

I'm working on a long-form article describing this project, complete with code samples, code review and screenshots. If you'd like to access an early draft of the content, enter your email below and I will get in touch.