Intermediate

35 mins

Logging

Learn how to implement comprehensive logging for your applications. This guide covers logging configuration, structured logging, storage, rotation, and analysis techniques.

Prerequisites

Understanding of application architecture
Familiarity with error handling
Basic knowledge of data formats (JSON, etc.)
Experience with Node.js applications

Logging Overview

Visual representation of the logging workflow from application events to analysis.

Logging Configuration

Set up a comprehensive logging system:

// Logging configuration
const loggingConfig = {
  // Log levels
  levels: {
    error: 0,
    warn: 1,
    info: 2,
    http: 3,
    verbose: 4,
    debug: 5,
    silly: 6
  },
  
  // Log formats
  formats: {
    simple: '{{timestamp}} {{level}}: {{message}}',
    detailed: '{{timestamp}} [{{level}}] {{label}}: {{message}} {{metadata}}',
    json: '{"timestamp":"{{timestamp}}","level":"{{level}}","message":"{{message}}","metadata":{{metadata}}}'
  },
  
  // Transport types
  transports: {
    console: {
      level: 'info',
      format: 'simple',
      colorize: true
    },
    file: {
      level: 'debug',
      format: 'detailed',
      filename: 'app.log',
      maxSize: '10m',
      maxFiles: 5
    },
    syslog: {
      level: 'error',
      format: 'detailed',
      host: 'logs.example.com',
      port: 514
    }
  },
  
  // Initialize logger
  createLogger(options = {}) {
    return {
      level: options.level || 'info',
      format: options.format || 'simple',
      transports: options.transports || ['console'],
      
      log(level, message, metadata = {}) {
        if (this.levels[level] > this.levels[this.level]) {
          return; // Skip logs above current level
        }
        
        const logEntry = {
          timestamp: new Date().toISOString(),
          level,
          message,
          metadata: JSON.stringify(metadata)
        };
        
        for (const transport of this.transports) {
          this.writeToTransport(transport, logEntry);
        }
      },
      
      error(message, metadata) {
        this.log('error', message, metadata);
      },
      
      warn(message, metadata) {
        this.log('warn', message, metadata);
      },
      
      info(message, metadata) {
        this.log('info', message, metadata);
      },
      
      debug(message, metadata) {
        this.log('debug', message, metadata);
      }
    };
  }
}

Structured Logging

Implement structured logging for better analysis:

// Structured logging implementation
const structuredLogger = {
  // Create structured log entry
  createLogEntry(level, message, context = {}) {
    return {
      timestamp: new Date().toISOString(),
      level,
      message,
      
      // Standard context fields
      service: context.service || process.env.SERVICE_NAME,
      environment: context.environment || process.env.NODE_ENV,
      version: context.version || process.env.APP_VERSION,
      
      // Request context if available
      request: context.request ? {
        id: context.request.id,
        method: context.request.method,
        path: context.request.path,
        ip: context.request.ip,
        userAgent: context.request.userAgent
      } : undefined,
      
      // User context if available
      user: context.user ? {
        id: context.user.id,
        type: context.user.type
      } : undefined,
      
      // Error context if available
      error: context.error ? {
        name: context.error.name,
        message: context.error.message,
        stack: context.error.stack,
        code: context.error.code
      } : undefined,
      
      // Additional context
      ...context.additional
    };
  },
  
  // Log middleware for HTTP requests
  requestLogMiddleware(options = {}) {
    return (req, res, next) => {
      // Generate request ID if not exists
      req.id = req.id || this.generateId();
      
      // Capture start time
      const startTime = Date.now();
      
      // Log request
      this.info('Request received', {
        request: {
          id: req.id,
          method: req.method,
          path: req.path,
          query: req.query,
          ip: req.ip,
          userAgent: req.get('user-agent')
        },
        user: req.user ? {
          id: req.user.id,
          type: req.user.type
        } : undefined
      });
      
      // Capture response
      const originalEnd = res.end;
      res.end = function(chunk, encoding) {
        // Restore original end
        res.end = originalEnd;
        
        // Calculate duration
        const duration = Date.now() - startTime;
        
        // Log response
        const level = res.statusCode >= 500 ? 'error' : 
                     res.statusCode >= 400 ? 'warn' : 'info';
        
        this.log(level, 'Request completed', {
          request: {
            id: req.id,
            method: req.method,
            path: req.path
          },
          response: {
            statusCode: res.statusCode,
            duration
          }
        });
        
        // Call original end
        return originalEnd.call(this, chunk, encoding);
      };
      
      next();
    };
  }
}

Log Storage and Rotation

Configure log storage, rotation, and retention:

// Log storage and rotation configuration
const logStorage = {
  // File storage configuration
  file: {
    async configure(options = {}) {
      return {
        directory: options.directory || 'logs',
        filename: options.filename || 'app-%DATE%.log',
        datePattern: options.datePattern || 'YYYY-MM-DD',
        maxSize: options.maxSize || '10m',
        maxFiles: options.maxFiles || '14d',
        
        async initialize() {
          // Create log directory if it doesn't exist
          if (!fs.existsSync(this.directory)) {
            fs.mkdirSync(this.directory, { recursive: true });
          }
          
          return this;
        }
      };
    }
  },
  
  // Log rotation
  rotation: {
    async configure(storage, options = {}) {
      return {
        storage,
        maxSize: options.maxSize || storage.maxSize,
        maxFiles: options.maxFiles || storage.maxFiles,
        compress: options.compress !== false,
        
        async rotate(logFile) {
          const stats = fs.statSync(logFile);
          const fileSize = stats.size / (1024 * 1024); // Convert to MB
          
          if (fileSize < this.parseSize(this.maxSize)) {
            return false; // No rotation needed
          }
          
          const timestamp = new Date().toISOString().replace(/:/g, '-');
          const rotatedFile = `${logFile}.${timestamp}`;
          
          // Rename current log file
          fs.renameSync(logFile, rotatedFile);
          
          // Compress if needed
          if (this.compress) {
            await this.compressFile(rotatedFile);
          }
          
          // Clean up old files
          await this.cleanupOldFiles();
          
          return true;
        },
        
        async compressFile(file) {
          const gzip = zlib.createGzip();
          const source = fs.createReadStream(file);
          const destination = fs.createWriteStream(`${file}.gz`);
          
          return new Promise((resolve, reject) => {
            source.pipe(gzip).pipe(destination)
              .on('finish', () => {
                fs.unlinkSync(file); // Remove original file
                resolve();
              })
              .on('error', reject);
          });
        }
      };
    }
  },
  
  // Log retention
  retention: {
    async configure(options = {}) {
      return {
        maxAge: options.maxAge || '30d',
        
        async applyRetention(directory, pattern) {
          const files = await this.getLogFiles(directory, pattern);
          const maxAgeDate = this.calculateMaxAgeDate();
          
          for (const file of files) {
            const stats = fs.statSync(file);
            if (stats.mtime < maxAgeDate) {
              fs.unlinkSync(file);
            }
          }
        },
        
        calculateMaxAgeDate() {
          const maxAge = this.parseMaxAge();
          return new Date(Date.now() - maxAge);
        }
      };
    }
  }
}

Log Analysis and Visualization

Set up log analysis and visualization tools:

// Log analysis configuration
const logAnalysis = {
  // Search and query
  search: {
    async query(options) {
      const {
        logSource,
        timeRange,
        levels,
        searchTerms,
        limit,
        offset
      } = options;
      
      // Build query
      const query = {
        bool: {
          must: [],
          filter: []
        }
      };
      
      // Add time range
      if (timeRange) {
        query.bool.filter.push({
          range: {
            timestamp: {
              gte: timeRange.start,
              lte: timeRange.end
            }
          }
        });
      }
      
      // Add log levels
      if (levels && levels.length > 0) {
        query.bool.filter.push({
          terms: {
            level: levels
          }
        });
      }
      
      // Add search terms
      if (searchTerms) {
        query.bool.must.push({
          query_string: {
            query: searchTerms,
            fields: ['message', 'error.message', 'request.path']
          }
        });
      }
      
      // Execute search
      return await this.executeSearch(logSource, query, limit, offset);
    }
  },
  
  // Aggregations and metrics
  aggregations: {
    async errorsByTime(timeRange, interval = '1h') {
      return await this.aggregate({
        timeRange,
        levels: ['error'],
        aggregation: {
          errors_over_time: {
            date_histogram: {
              field: 'timestamp',
              interval
            }
          }
        }
      });
    },
    
    async topErrorPaths(timeRange, limit = 10) {
      return await this.aggregate({
        timeRange,
        levels: ['error'],
        aggregation: {
          top_paths: {
            terms: {
              field: 'request.path',
              size: limit
            }
          }
        }
      });
    },
    
    async responseTimeStats(timeRange) {
      return await this.aggregate({
        timeRange,
        aggregation: {
          response_time_stats: {
            stats: {
              field: 'response.duration'
            }
          }
        }
      });
    }
  },
  
  // Visualization configuration
  visualization: {
    createDashboard(name, description) {
      return {
        name,
        description,
        timeRange: {
          start: 'now-24h',
          end: 'now'
        },
        refreshInterval: '1m',
        panels: []
      };
    },
    
    addTimeSeriesPanel(dashboard, title, metric, options = {}) {
      const panel = {
        id: this.generateId(),
        title,
        type: 'time_series',
        metric,
        options: {
          interval: options.interval || '1h',
          visualization: options.visualization || 'line',
          ...options
        }
      };
      
      dashboard.panels.push(panel);
      return dashboard;
    }
  }
}

Best Practices

Log Content

Best practices for log content:

Use structured logging
Include context information
Standardize log formats
Avoid sensitive data

Performance

Optimize logging performance:

Use asynchronous logging
Implement log buffering
Configure appropriate levels
Monitor logging overhead

Security

Secure your logging system:

Encrypt sensitive logs
Implement access controls
Protect log integrity
Regular log reviews

Log Levels and When to Use Them

ERROR

Critical

Use for errors that prevent the application from functioning correctly. These require immediate attention.

Examples:

Database connection failures
Unhandled exceptions
API integration failures
System resource exhaustion

WARN

Warning

Use for potentially harmful situations that don't prevent the application from working but should be addressed.

Examples:

Deprecated API usage
Retry attempts
Fallback to secondary systems
Configuration issues

INFO

Informational

Use for general information about application operation and important lifecycle events.

Examples:

Application startup/shutdown
User authentication
Configuration loading
Significant business events

DEBUG

Detailed

Use for detailed information useful during development and troubleshooting. Not typically enabled in production.

Examples:

Function entry/exit points
Variable values
API request/response details
SQL queries

Structured Logging Examples

Error Log Example

{
  "timestamp": "2025-05-15T14:32:45.123Z",
  "level": "error",
  "message": "Failed to connect to database",
  "service": "user-service",
  "environment": "production",
  "error": {
    "name": "ConnectionError",
    "message": "Connection refused",
    "code": "ECONNREFUSED",
    "stack": "ConnectionError: Connection refused\\n    at Database.connect (/app/src/database.js:45:23)\\n    at async UserService.getUsers (/app/src/services/user-service.js:12:5)"
  },
  "metadata": {
    "host": "db.example.com",
    "port": 5432,
    "database": "users",
    "retryAttempt": 3
  }
}

Request Log Example

{
  "timestamp": "2025-05-15T14:33:12.456Z",
  "level": "info",
  "message": "Request completed",
  "service": "api-gateway",
  "environment": "production",
  "request": {
    "id": "req_7f8d9e2a3b1c",
    "method": "POST",
    "path": "/api/users",
    "ip": "192.168.1.1",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
  },
  "response": {
    "statusCode": 201,
    "duration": 125
  },
  "user": {
    "id": "usr_1a2b3c4d5e",
    "type": "admin"
  }
}

Common Issues

Performance Impact

Common performance issues:

Synchronous logging blocking operations
Excessive logging volume
Inefficient log serialization
Disk I/O bottlenecks

Storage Issues

Storage-related challenges:

Disk space exhaustion
Log rotation failures
Permission problems
Network transport failures

Next Steps

Now that you understand logging, explore these related topics: