Advanced

50 mins

Threat Detection

Learn how to implement comprehensive threat detection systems using threat intelligence, behavioral analysis, signature-based detection, and event correlation techniques.

Prerequisites

Understanding of security concepts and threats
Experience with security monitoring
Knowledge of network and system architecture
Familiarity with data analysis techniques

Threat Detection Overview

Visual representation of the threat detection process and key components.

Threat Intelligence Integration

Set up threat intelligence feeds and integration:

// Threat intelligence configuration
const threatIntelligence = {
  // Intelligence sources
  sources: {
    osint: {
      name: 'Open Source Intelligence',
      feeds: [
        { name: 'AlienVault OTX', url: 'https://otx.alienvault.com/api/v1/indicators' },
        { name: 'MISP', url: 'https://misp.example.org/api/indicators' },
        { name: 'Abuse.ch', url: 'https://abuse.ch/api/indicators' }
      ],
      updateInterval: 3600 // seconds
    },
    commercial: {
      name: 'Commercial Intelligence',
      feeds: [
        { name: 'Recorded Future', url: 'https://api.recordedfuture.com/v2/indicators' },
        { name: 'Crowdstrike', url: 'https://api.crowdstrike.com/intel/indicators' }
      ],
      updateInterval: 7200 // seconds
    },
    internal: {
      name: 'Internal Intelligence',
      feeds: [
        { name: 'Security Incidents', url: 'internal://security/incidents' },
        { name: 'Threat Hunting', url: 'internal://security/hunting' }
      ],
      updateInterval: 1800 // seconds
    }
  },

  // Indicator types
  indicatorTypes: {
    ip: { name: 'IP Address', ttl: 604800 }, // 7 days
    domain: { name: 'Domain Name', ttl: 1209600 }, // 14 days
    url: { name: 'URL', ttl: 604800 }, // 7 days
    hash: { name: 'File Hash', ttl: 2592000 }, // 30 days
    email: { name: 'Email Address', ttl: 1209600 } // 14 days
  },

  async fetchIntelligence() {
    const intelligence = {
      timestamp: new Date(),
      indicators: {}
    };
    
    // Fetch from all sources
    for (const [sourceKey, source] of Object.entries(this.sources)) {
      console.log(`Fetching intelligence from ${source.name}`);
      
      for (const feed of source.feeds) {
        try {
          const indicators = await this.fetchFromFeed(feed.url);
          
          // Process and categorize indicators
          for (const indicator of indicators) {
            const type = this.categorizeIndicator(indicator);
            
            if (!type) continue; // Skip if unknown type
            
            if (!intelligence.indicators[type]) {
              intelligence.indicators[type] = [];
            }
            
            intelligence.indicators[type].push({
              value: indicator.value,
              source: `${source.name}:${feed.name}`,
              firstSeen: indicator.firstSeen || new Date(),
              lastSeen: indicator.lastSeen || new Date(),
              confidence: indicator.confidence || 'medium',
              context: indicator.context || {},
              ttl: this.indicatorTypes[type].ttl
            });
          }
        } catch (error) {
          console.error(`Error fetching from ${feed.name}: ${error.message}`);
        }
      }
    }
    
    return intelligence;
  }
}

Behavioral Analysis

Implement behavioral analysis for anomaly detection:

// Behavioral analysis system
const behavioralAnalysis = {
  // Behavior models
  models: {
    userActivity: {
      name: 'User Activity Model',
      features: [
        'login_frequency',
        'session_duration',
        'access_patterns',
        'resource_usage',
        'geographic_location'
      ],
      thresholds: {
        login_frequency: { min: 0, max: 20, timeWindow: 86400 }, // per day
        session_duration: { min: 60, max: 28800 }, // 1 min to 8 hours
        geographic_velocity: { maxSpeed: 500 } // km/h
      }
    },
    
    networkTraffic: {
      name: 'Network Traffic Model',
      features: [
        'traffic_volume',
        'protocol_distribution',
        'connection_patterns',
        'packet_sizes',
        'destination_ips'
      ],
      thresholds: {
        traffic_volume: { maxDeviation: 3 }, // standard deviations
        new_destinations: { max: 50, timeWindow: 3600 }, // per hour
        connection_rate: { max: 1000, timeWindow: 60 } // per minute
      }
    },
    
    systemActivity: {
      name: 'System Activity Model',
      features: [
        'cpu_usage',
        'memory_usage',
        'disk_activity',
        'process_creation',
        'file_access'
      ],
      thresholds: {
        process_creation: { max: 100, timeWindow: 60 }, // per minute
        file_access_rate: { max: 1000, timeWindow: 60 }, // per minute
        resource_usage: { maxDeviation: 3 } // standard deviations
      }
    }
  },
  
  // Analyze behavior
  async analyzeBehavior(entity, model, data) {
    console.log(`Analyzing ${entity} using ${model.name}`);
    
    // Get historical data for baseline
    const baseline = await this.getBaseline(entity, model);
    
    // Calculate anomaly scores
    const scores = {};
    for (const feature of model.features) {
      if (data[feature]) {
        scores[feature] = await this.calculateAnomalyScore(
          data[feature],
          baseline[feature],
          model.thresholds[feature]
        );
      }
    }
    
    // Calculate overall anomaly score
    const overallScore = this.calculateOverallScore(scores);
    
    return {
      entity,
      timestamp: new Date(),
      model: model.name,
      scores,
      overallScore,
      isAnomaly: overallScore > this.anomalyThreshold
    };
  },
  
  // Calculate anomaly score for a feature
  async calculateAnomalyScore(value, baseline, threshold) {
    if (!baseline || !baseline.mean || !baseline.stdDev) {
      return 0; // Not enough data for baseline
    }
    
    // Z-score calculation
    const zScore = Math.abs((value - baseline.mean) / baseline.stdDev);
    
    // Apply threshold if available
    if (threshold) {
      if (threshold.min !== undefined && value < threshold.min) {
        return Math.max(zScore, 1); // Ensure minimum anomaly score
      }
      
      if (threshold.max !== undefined && value > threshold.max) {
        return Math.max(zScore, 1); // Ensure minimum anomaly score
      }
      
      if (threshold.maxDeviation !== undefined && zScore > threshold.maxDeviation) {
        return zScore;
      }
    }
    
    return zScore;
  }
}

Signature-Based Detection

Configure signature-based threat detection:

// Signature-based detection system
const signatureDetection = {
  // Signature categories
  categories: {
    network: {
      name: 'Network Signatures',
      types: ['ip', 'domain', 'url', 'protocol', 'packet']
    },
    host: {
      name: 'Host Signatures',
      types: ['file', 'process', 'registry', 'memory']
    },
    application: {
      name: 'Application Signatures',
      types: ['api', 'authentication', 'database', 'input']
    }
  },
  
  // Signature database
  signatures: {
    async load() {
      const db = {
        network: await this.loadNetworkSignatures(),
        host: await this.loadHostSignatures(),
        application: await this.loadApplicationSignatures()
      };
      
      console.log(`Loaded ${Object.values(db).flat().length} signatures`);
      return db;
    },
    
    async update() {
      console.log('Updating signature database');
      
      // Fetch latest signatures from sources
      const newSignatures = await this.fetchLatestSignatures();
      
      // Merge with existing signatures
      const updated = await this.mergeSignatures(newSignatures);
      
      console.log(`Updated signature database: ${updated.added} added, ${updated.updated} updated, ${updated.removed} removed`);
      
      return updated;
    }
  },
  
  // Match events against signatures
  async matchSignatures(events, category) {
    const signatures = await this.signatures.load();
    const categorySignatures = signatures[category];
    
    if (!categorySignatures) {
      throw new Error(`Unknown signature category: ${category}`);
    }
    
    const matches = [];
    
    for (const event of events) {
      for (const signature of categorySignatures) {
        if (await this.matchSignature(event, signature)) {
          matches.push({
            event,
            signature,
            timestamp: new Date(),
            confidence: signature.confidence || 'medium'
          });
        }
      }
    }
    
    return matches;
  },
  
  // Match a single event against a signature
  async matchSignature(event, signature) {
    // Simple exact match
    if (signature.type === 'exact' && signature.value === event.value) {
      return true;
    }
    
    // Regex match
    if (signature.type === 'regex' && new RegExp(signature.pattern).test(event.value)) {
      return true;
    }
    
    // YARA match for files
    if (signature.type === 'yara' && event.data && signature.rules) {
      return await this.matchYara(event.data, signature.rules);
    }
    
    // Suricata match for network traffic
    if (signature.type === 'suricata' && event.packet && signature.rule) {
      return await this.matchSuricata(event.packet, signature.rule);
    }
    
    return false;
  }
}

Correlation and Analysis

Implement event correlation and analysis:

// Event correlation engine
const correlationEngine = {
  // Correlation rules
  rules: {
    async load() {
      return {
        bruteForce: {
          name: 'Brute Force Detection',
          description: 'Detects multiple failed login attempts',
          conditions: [
            { event: 'authentication', result: 'failure', threshold: 5, timeWindow: 300 }
          ],
          severity: 'high'
        },
        
        dataExfiltration: {
          name: 'Data Exfiltration Detection',
          description: 'Detects potential data exfiltration',
          conditions: [
            { event: 'file_access', operation: 'read', threshold: 10, timeWindow: 60 },
            { event: 'network_connection', direction: 'outbound', threshold: 1, timeWindow: 60 }
          ],
          severity: 'critical'
        },
        
        privilegeEscalation: {
          name: 'Privilege Escalation Detection',
          description: 'Detects potential privilege escalation',
          conditions: [
            { event: 'process_execution', parent: 'user_process', child: 'system_process' },
            { event: 'permission_change', target: 'user', operation: 'elevate' }
          ],
          severity: 'critical'
        },
        
        lateralMovement: {
          name: 'Lateral Movement Detection',
          description: 'Detects potential lateral movement',
          conditions: [
            { event: 'authentication', source: 'internal', target: 'internal', newHost: true },
            { event: 'network_connection', protocol: 'smb|rdp|ssh', direction: 'outbound', newDestination: true }
          ],
          severity: 'high'
        }
      };
    }
  },
  
  // Correlate events
  async correlateEvents(events) {
    const rules = await this.rules.load();
    const alerts = [];
    
    for (const [ruleId, rule] of Object.entries(rules)) {
      const matches = await this.matchRule(events, rule);
      
      if (matches.length > 0) {
        alerts.push({
          ruleId,
          ruleName: rule.name,
          description: rule.description,
          severity: rule.severity,
          timestamp: new Date(),
          matches,
          context: await this.gatherContext(matches)
        });
      }
    }
    
    return alerts;
  },
  
  // Match events against a rule
  async matchRule(events, rule) {
    const matches = [];
    
    // Group events by type
    const eventsByType = {};
    for (const event of events) {
      if (!eventsByType[event.type]) {
        eventsByType[event.type] = [];
      }
      eventsByType[event.type].push(event);
    }
    
    // Check each condition
    for (const condition of rule.conditions) {
      const typeEvents = eventsByType[condition.event] || [];
      
      // Filter events by condition properties
      const filteredEvents = typeEvents.filter(event => {
        for (const [key, value] of Object.entries(condition)) {
          if (key === 'event' || key === 'threshold' || key === 'timeWindow') {
            continue; // Skip non-event properties
          }
          
          // Handle special conditions
          if (key === 'newHost' && value === true) {
            return this.isNewHost(event);
          }
          
          if (key === 'newDestination' && value === true) {
            return this.isNewDestination(event);
          }
          
          // Handle regex values
          if (typeof value === 'string' && value.includes('|')) {
            const regex = new RegExp(`^${value}$`);
            if (!regex.test(event[key])) {
              return false;
            }
          } 
          // Handle exact match
          else if (event[key] !== value) {
            return false;
          }
        }
        
        return true;
      });
      
      // Check threshold if specified
      if (condition.threshold) {
        if (filteredEvents.length < condition.threshold) {
          return []; // Threshold not met
        }
        
        // Check time window if specified
        if (condition.timeWindow) {
          const windowStart = new Date(Date.now() - condition.timeWindow * 1000);
          const eventsInWindow = filteredEvents.filter(event => 
            new Date(event.timestamp) >= windowStart
          );
          
          if (eventsInWindow.length < condition.threshold) {
            return []; // Threshold not met within time window
          }
          
          matches.push(...eventsInWindow);
        } else {
          matches.push(...filteredEvents);
        }
      } else {
        matches.push(...filteredEvents);
      }
    }
    
    return matches;
  }
}

Response Automation

Set up automated responses to detected threats:

// Automated response system
const responseAutomation = {
  // Response actions
  actions: {
    blockIP: {
      name: 'Block IP Address',
      description: 'Block an IP address at the firewall',
      async execute(params) {
        console.log(`Blocking IP address: ${params.ip}`);
        
        // Validate IP address
        if (!this.isValidIP(params.ip)) {
          throw new Error(`Invalid IP address: ${params.ip}`);
        }
        
        // Check if already blocked
        const isBlocked = await this.checkIPBlocked(params.ip);
        if (isBlocked) {
          return { status: 'already_blocked', ip: params.ip };
        }
        
        // Add to blocklist
        await this.addToBlocklist(params.ip, {
          reason: params.reason || 'Automated response',
          duration: params.duration || 3600, // 1 hour default
          source: params.source || 'threat_detection'
        });
        
        return { status: 'blocked', ip: params.ip };
      }
    },
    
    isolateHost: {
      name: 'Isolate Host',
      description: 'Isolate a host from the network',
      async execute(params) {
        console.log(`Isolating host: ${params.host}`);
        
        // Validate host
        if (!this.isValidHost(params.host)) {
          throw new Error(`Invalid host: ${params.host}`);
        }
        
        // Check if already isolated
        const isIsolated = await this.checkHostIsolated(params.host);
        if (isIsolated) {
          return { status: 'already_isolated', host: params.host };
        }
        
        // Isolate host
        await this.isolateHost(params.host, {
          reason: params.reason || 'Automated response',
          duration: params.duration || 3600, // 1 hour default
          source: params.source || 'threat_detection'
        });
        
        return { status: 'isolated', host: params.host };
      }
    },
    
    terminateSession: {
      name: 'Terminate Session',
      description: 'Terminate a user session',
      async execute(params) {
        console.log(`Terminating session: ${params.sessionId}`);
        
        // Validate session
        if (!this.isValidSession(params.sessionId)) {
          throw new Error(`Invalid session ID: ${params.sessionId}`);
        }
        
        // Terminate session
        await this.terminateSession(params.sessionId, {
          reason: params.reason || 'Automated response',
          source: params.source || 'threat_detection'
        });
        
        return { status: 'terminated', sessionId: params.sessionId };
      }
    },
    
    createIncident: {
      name: 'Create Security Incident',
      description: 'Create a security incident ticket',
      async execute(params) {
        console.log('Creating security incident');
        
        // Create incident
        const incident = await this.createIncident({
          title: params.title || 'Security Incident',
          description: params.description || 'Automated incident creation',
          severity: params.severity || 'high',
          source: params.source || 'threat_detection',
          assignee: params.assignee,
          artifacts: params.artifacts || []
        });
        
        return { status: 'created', incidentId: incident.id };
      }
    }
  },
  
  // Response playbooks
  playbooks: {
    async load() {
      return {
        bruteForceResponse: {
          name: 'Brute Force Response',
          description: 'Automated response to brute force attacks',
          trigger: 'bruteForce',
          actions: [
            {
              action: 'blockIP',
              params: {
                ip: '{{source.ip}}',
                reason: 'Brute force attack detected',
                duration: 7200 // 2 hours
              }
            },
            {
              action: 'createIncident',
              params: {
                title: 'Brute Force Attack',
                description: 'Multiple failed login attempts detected from {{source.ip}}',
                severity: 'high',
                artifacts: [
                  { type: 'ip', value: '{{source.ip}}' },
                  { type: 'user', value: '{{target.user}}' }
                ]
              }
            }
          ]
        },
        
        dataExfiltrationResponse: {
          name: 'Data Exfiltration Response',
          description: 'Automated response to potential data exfiltration',
          trigger: 'dataExfiltration',
          actions: [
            {
              action: 'isolateHost',
              params: {
                host: '{{source.host}}',
                reason: 'Potential data exfiltration detected',
                duration: 3600 // 1 hour
              }
            },
            {
              action: 'terminateSession',
              params: {
                sessionId: '{{source.sessionId}}',
                reason: 'Potential data exfiltration detected'
              }
            },
            {
              action: 'createIncident',
              params: {
                title: 'Potential Data Exfiltration',
                description: 'Suspicious data transfer detected from {{source.host}}',
                severity: 'critical',
                assignee: 'security-team',
                artifacts: [
                  { type: 'host', value: '{{source.host}}' },
                  { type: 'user', value: '{{source.user}}' },
                  { type: 'destination', value: '{{destination.ip}}' }
                ]
              }
            }
          ]
        }
      };
    }
  },
  
  // Execute response
  async executeResponse(alert) {
    console.log(`Executing automated response for alert: ${alert.ruleName}`);
    
    // Load playbooks
    const playbooks = await this.playbooks.load();
    
    // Find matching playbook
    const playbook = playbooks[alert.ruleId];
    
    if (!playbook) {
      console.log(`No playbook found for rule: ${alert.ruleId}`);
      return null;
    }
    
    console.log(`Executing playbook: ${playbook.name}`);
    
    // Execute actions
    const results = [];
    
    for (const actionConfig of playbook.actions) {
      const action = this.actions[actionConfig.action];
      
      if (!action) {
        console.error(`Unknown action: ${actionConfig.action}`);
        continue;
      }
      
      try {
        // Process parameter templates
        const params = this.processTemplates(actionConfig.params, alert);
        
        // Execute action
        const result = await action.execute(params);
        results.push({
          action: actionConfig.action,
          status: 'success',
          result
        });
      } catch (error) {
        console.error(`Error executing action ${actionConfig.action}: ${error.message}`);
        results.push({
          action: actionConfig.action,
          status: 'error',
          error: error.message
        });
      }
    }
    
    return {
      playbook: playbook.name,
      timestamp: new Date(),
      alert: alert.id,
      actions: results
    };
  }
}

Best Practices

Data Collection

Best practices for threat data collection:

Collect from diverse sources
Ensure data quality
Implement proper retention
Maintain context

Detection Strategy

Effective threat detection approach:

Layer multiple methods
Tune for low false positives
Regular rule updates
Continuous improvement

Response Planning

Plan effective responses:

Automate common responses
Define escalation paths
Document procedures
Regular testing

Detection Methods Comparison

Method	Strengths	Weaknesses	Best For
Signature-Based	Low false positives Easy to implement Clear detection logic	Only detects known threats Requires regular updates Easily evaded by variants	Known malware, specific attack patterns
Behavioral Analysis	Detects unknown threats Harder to evade Adapts to changing patterns	Higher false positives Requires baseline period More resource intensive	Zero-day attacks, insider threats, APTs
Threat Intelligence	Leverages external knowledge Provides context Proactive protection	Quality varies by source Can become outdated May not be relevant to your environment	Known threat actors, campaigns, IOCs
Correlation Analysis	Connects related events Reduces alert fatigue Provides attack context	Complex to implement Requires diverse data sources Rule maintenance overhead	Multi-stage attacks, complex threats

Common Threat Scenarios

Credential Stuffing Attack

An attacker uses leaked username/password pairs from other breaches to attempt to access user accounts.

Detection Approach:

Behavioral: Unusual number of failed logins across multiple accounts
Signature: Known attack patterns in login attempts
Intelligence: IPs associated with credential stuffing campaigns

Response Actions:

Block source IP addresses
Implement CAPTCHA after failed attempts
Force password reset for affected accounts
Alert security team

Data Exfiltration

An insider or compromised account attempts to extract sensitive data from the organization.

Detection Approach:

Behavioral: Unusual data access patterns or volumes
Correlation: Database access followed by large outbound transfer
Signature: Known exfiltration tools or techniques

Response Actions:

Terminate user session
Isolate affected systems
Block outbound connections
Create high-priority incident

Lateral Movement

An attacker moves through the network after gaining initial access, attempting to reach sensitive systems.

Detection Approach:

Behavioral: Unusual authentication patterns between systems
Correlation: Initial compromise followed by internal scanning
Signature: Known lateral movement tools (PsExec, WMI, etc.)

Response Actions:

Isolate affected systems
Reset compromised credentials
Block internal movement
Initiate incident response

Common Challenges

False Positives

Strategies for reducing false positives:

Tune detection thresholds
Implement multi-factor detection
Use contextual information
Regular rule review and refinement

Alert Fatigue

Approaches to combat alert fatigue:

Alert prioritization
Alert correlation
Automated triage
Focused alerting policies

Next Steps

Now that you understand threat detection, explore these related topics: