Skip to main content

CI Relationship System Documentation

Overview​

The CI Relationship System in NopeSight v3 provides intelligent mapping of network connections between Configuration Items (CIs) to automatically discover and maintain service dependencies. The system uses a bidirectional relationship architecture that ensures data consistency and accurate direction determination.

Core Concepts​

Bidirectional Relationships​

Unlike traditional systems that create separate relationship records from each CI's perspective, NopeSight uses a single relationship object for each connection pair. This approach:

  • Prevents duplicate relationships between the same CIs
  • Maintains consistency across multiple scans
  • Reduces data redundancy in the CMDB
  • Provides dual perspectives through sourceView and targetView attributes

Direction Determination​

The system uses a hierarchical approach to determine the correct direction of relationships:

  1. ListeningService Registry (Highest Priority)
  2. Connection State Analysis (LISTEN state)
  3. Well-Known Port Detection
  4. Port Range Classification (Ephemeral vs Registered)
  5. Deterministic Fallback (Lower port number)
  6. Alphabetical Ordering (Last resort)

Architecture​

Data Model​

CIRelationship {
  source: ObjectId,           // CI initiating connection
  target: ObjectId,           // CI receiving connection
  sourceType: String,         // 'CI' or 'UnknownDevice'
  targetType: String,         // 'CI' or 'UnknownDevice'
  sourceIdentifier: String,   // For unknown devices (IP:port)
  targetIdentifier: String,   // For unknown devices (IP:port)
  type: ObjectId,            // Relationship type (Connected To)
  attributes: Map {
    sourceView: {            // Source CI's perspective
      process: String,
      pid: Number,
      localPorts: [Number],
      remotePorts: [Number],
      connectionCount: Number,
      lastUpdated: Date
    },
    targetView: {            // Target CI's perspective
      process: String,
      pid: Number,
      localPorts: [Number],
      remotePorts: [Number],
      connectionCount: Number,
      lastUpdated: Date
    },
    servicePorts: [Number],  // Service ports involved
    protocol: String,        // TCP/UDP
    confidence: String,      // high/medium/low/guess
    directionReason: String, // Explanation of direction
    bidirectional: Boolean,  // For peer-to-peer connections
    lastUpdatedFrom: String  // Last CI that updated this
  }
}

Processing Flow​

Direction Determination Algorithm​

1. ListeningService Check​

The system first checks if either port is registered as a listening service:

// SQL Server custom ports example
ListeningService {
  name: "SQL Server Custom Instance",
  ports: [
    { port: 1440, protocol: "TCP" },
    { port: 1488, protocol: "TCP" }
  ],
  tenant: ObjectId("..."),
  isActive: true
}

Result: CI with listening port becomes the target (receiver)

2. Connection State Analysis​

For connections with state information:

  • State = 2 (LISTEN): CI is the target
  • State = 5 (ESTABLISHED): Further analysis needed
  • State = 11 (TIME_WAIT): CI was likely the source

3. Well-Known Ports​

Common service ports indicate the server side:

const WELL_KNOWN_PORTS = [
  21,    // FTP
  22,    // SSH
  80,    // HTTP
  443,   // HTTPS
  1433,  // SQL Server
  1521,  // Oracle
  3306,  // MySQL
  5432,  // PostgreSQL
  // ... etc
]

Result: CI with well-known port becomes the target

4. Port Range Classification​

  • Registered Ports (1024-49151): Likely services
  • Ephemeral Ports (49152-65535): Client connections

Example:

  • CI1 using port 50123 (ephemeral) → CI2 using port 8080 (registered)
  • Direction: CI1 is source, CI2 is target

5. Non-Listener Connections​

For peer-to-peer or RPC connections where neither side is listening:

// Example: RPC connection
CI1: Port 48765 → CI2: Port 49123
Both ephemeral, no listener detected

// Fallback rules applied:
1. Lower port number (48765) assumed to be service
2. CI2 becomes source, CI1 becomes target
3. Confidence marked as "low" or "guess"

Implementation Details​

Key Functions​

determineSourceAndTarget(ci1, ci2, conn1, conn2, tenant)​

Main direction determination logic implementing the hierarchical rules.

Parameters:

  • ci1: First CI object
  • ci2: Second CI object (can be null)
  • conn1: Connection data from ci1's perspective
  • conn2: Connection data from ci2's perspective (optional)
  • tenant: Tenant ID for service lookup

Returns:

{
  source: CI,           // Source CI object
  target: CI,           // Target CI object
  confidence: String,   // 'high'|'medium'|'low'|'guess'
  reason: String,       // Explanation
  bidirectional: Boolean // True for peer connections
}

findOrCreateRelationship(sourceCI, targetCI, type, identifier)​

Finds existing relationship or creates new one, ensuring single record per connection pair.

Key Logic:

  • Searches for relationship in both directions
  • Corrects direction if existing relationship is reversed
  • Handles unknown devices with identifiers

updateRelationshipView(relationship, scanningCI, connectionData, isSource, appInfo)​

Updates the appropriate view (sourceView or targetView) based on scanning CI's role.

Features:

  • Aggregates multiple connections to same endpoint
  • Preserves historical data (ports, processes)
  • Maintains connection counts
  • Stores recent connection details (last 100)
  • Enriches with application information
  • Adds software family metadata

Connection History:

connections: [
  {
    localAddress: "10.1.1.50",
    remoteAddress: "10.1.1.100",
    localPort: 52456,
    remotePort: 1440,
    protocol: "TCP",
    state: 5,  // ESTABLISHED
    process: "java.exe",
    pid: 12345
  },
  // ... up to 100 recent connections
]

Unknown Device Handling​

When a remote CI cannot be found:

// Unknown device representation
{
  sourceType: 'UnknownDevice',
  sourceIdentifier: '192.168.1.100:3306',  // IP:port format
  target: knownCI._id
}

This allows tracking connections to:

  • External services
  • Unscanned devices
  • Temporary connections
  • Cloud services

Real-World Scenarios​

Scenario 1: Database Connection​

Application Server (KTPRDISOMS) → Database Server (KCTSDB01)
Port 52456 → Port 1440 (SQL Server)

Analysis:
1. Port 1440 found in ListeningService
2. KCTSDB01 identified as target
3. Relationship: KTPRDISOMS (source) → KCTSDB01 (target)
4. Confidence: high

Scenario 2: Web Service​

Load Balancer → Web Server
Port 54321 → Port 443 (HTTPS)

Analysis:
1. Port 443 is well-known HTTPS port
2. Web Server identified as target
3. Relationship: Load Balancer (source) → Web Server (target)
4. Confidence: high

Scenario 3: RPC/Peer Connection​

Service A ↔ Service B
Port 45123 ↔ Port 45678 (both RPC)

Analysis:
1. No listening service detected
2. Both ports in registered range
3. Lower port (45123) assumed as service
4. Relationship: Service B (source) → Service A (target)
5. Confidence: guess
6. Marked as bidirectional

Scenario 4: Unknown External Service​

Internal Server → External API
Port 53421 → 34.102.136.180:443

Analysis:
1. External IP not in CMDB
2. Port 443 indicates HTTPS service
3. Creates UnknownDevice target
4. Relationship: Internal Server (source) → Unknown:34.102.136.180:443 (target)
5. Confidence: medium

Configuration​

ListeningService Management​

Add custom service ports for accurate direction detection:

// Via API or database
POST /api/listening-services
{
  "name": "Custom Application",
  "ports": [
    { "port": 9090, "protocol": "TCP" }
  ],
  "tenant": "tenant-id",
  "isActive": true
}

Relationship Type Configuration​

The system uses "Connected To" relationship type by default:

{
  name: "Connected To",
  description: "Network connection between CIs",
  category: "Dependency",
  isActive: true
}

Troubleshooting​

Common Issues​

1. Incorrect Relationship Direction​

Symptom: Database showing as source instead of target

Solution:

  • Verify port is in ListeningService registry
  • Check connection state values
  • Review confidence level in relationship attributes

2. Duplicate Relationships​

Symptom: Multiple relationships between same CIs

Solution:

  • Run cleanup script to merge duplicates
  • Verify both CIs update same relationship object
  • Check tenant isolation isn't creating separate records

3. Unknown Device Proliferation​

Symptom: Many UnknownDevice entries

Solution:

  • Ensure network discovery covers all devices
  • Check IP resolution in ServerNetworkAdapter
  • Verify tenant boundaries aren't blocking CI discovery

Debugging​

Enable detailed logging:

// In cirelation_processor.js
logger.debug(`Direction determined: ${direction.reason}`, {
  source: direction.source?.name,
  target: direction.target?.name,
  confidence: direction.confidence
});

Check relationship attributes for direction reasoning:

db.cirelationships.findOne({ _id: relationshipId })
// Review attributes.directionReason
// Check attributes.confidence

Migration from Previous Version​

Identifying Duplicates​

// MongoDB query to find potential duplicates
db.cirelationships.aggregate([
  {
    $group: {
      _id: {
        ci1: { $cond: [{ $lt: ["$source", "$target"] }, "$source", "$target"] },
        ci2: { $cond: [{ $lt: ["$source", "$target"] }, "$target", "$source"] }
      },
      count: { $sum: 1 },
      relationships: { $push: "$_id" }
    }
  },
  { $match: { count: { $gt: 1 } } }
])

Cleanup Script​

// Merge duplicate relationships
async function cleanupDuplicateRelationships() {
  const duplicates = await findDuplicates();

  for (const dup of duplicates) {
    const [keep, ...remove] = dup.relationships;

    // Merge views from duplicates
    await mergeRelationshipViews(keep, remove);

    // Delete duplicates
    await CIRelationship.deleteMany({
      _id: { $in: remove }
    });
  }
}

Best Practices​

1. Service Registration​

Always register custom service ports in ListeningService:

  • Improves direction accuracy
  • Reduces "guess" confidence relationships
  • Helps with service discovery

2. Scan Frequency​

Regular scanning ensures:

  • Both sides update relationship views
  • Connection state changes are captured
  • Unknown devices get identified when scanned

3. Tenant Isolation​

Maintain proper tenant boundaries:

  • Services should be in same tenant as their dependencies
  • Cross-tenant relationships need special handling
  • Global services can use null tenant

4. Performance Optimization​

Caching Strategy​

Port Cache:

  • 5-minute TTL for well-known ports
  • Tenant-specific cache entries
  • Reduces ListeningService queries

Application Cache (recommended):

// Implement in-memory cache for frequent lookups
const APP_CACHE = new Map();
const CACHE_TTL = 10 * 60 * 1000; // 10 minutes

Processing Optimizations​

  1. Skip Rule Processing:

    relationship._skipRuleProcessing = true;

    Bypasses unnecessary post-save hooks

  2. Connection Aggregation:

    • Groups by remote address
    • Finds most common port combinations
    • Reduces duplicate processing

  3. Batch Operations:

    • Process multiple CIs in parallel
    • Use MongoDB bulk operations for cleanup

  4. Query Optimization:

    // Indexes needed:
    db.cirelationships.createIndex({ source: 1, target: 1 })
    db.cirelationships.createIndex({ "attributes.servicePorts": 1 })
    db.listeningservices.createIndex({ "ports.port": 1, tenant: 1 })

Statistics and Monitoring​

{
  relationshipsCreated: 0,
  relationshipsUpdated: 0,
  unknownDevicesCreated: 0,
  unknownDevicesUpdated: 0,
  errors: 0,
  duplicatesFound: 0,
  duplicatesMerged: 0,
  skippedConnections: 0
}

Use these metrics to:

  • Monitor processing efficiency
  • Identify data quality issues
  • Track unknown device discovery
  • Measure deduplication effectiveness

API Reference​

Relationship Query API​

// Get all relationships for a CI
GET /api/ci-relationships?ci=<ci-id>

// Get relationships by direction
GET /api/ci-relationships?source=<ci-id>
GET /api/ci-relationships?target=<ci-id>

// Get relationships with confidence filter
GET /api/ci-relationships?ci=<ci-id>&confidence=high

Response Format​

{
  "_id": "relationship-id",
  "source": {
    "_id": "ci-id",
    "name": "Server-A"
  },
  "target": {
    "_id": "ci-id",
    "name": "Server-B"
  },
  "type": {
    "name": "Connected To"
  },
  "attributes": {
    "servicePorts": [443],
    "protocol": "TCP",
    "confidence": "high",
    "directionReason": "Well-known port: 443 on ci2",
    "sourceView": {
      "localPorts": [54321],
      "remotePorts": [443],
      "connectionCount": 5
    },
    "targetView": {
      "localPorts": [443],
      "remotePorts": [54321],
      "connectionCount": 5
    }
  }
}

Future Enhancements​

Planned Features​

  1. Machine Learning Direction Detection

    • Train model on confirmed relationships
    • Improve accuracy for ambiguous connections
    • Pattern recognition for complex services

  2. Service Discovery Enhancement

    • Automatic service type detection
    • Protocol analysis beyond port numbers
    • Application fingerprinting

  3. Relationship Lifecycle Management

    • Automatic cleanup of stale relationships
    • Historical relationship tracking
    • Change impact analysis

  4. Cross-Tenant Relationships

    • Secure cross-tenant dependency mapping
    • Service provider/consumer models
    • Multi-tenant service catalogs