Skip to content

Component Manager API Reference

The Heritage Data Processor Component Manager API provides comprehensive endpoints for managing pipeline components, including installation, updates, configuration, and health monitoring.

Base URL

All endpoints are prefixed with /pipeline_components.

Component Discovery & Listing

Get All Pipeline Components

Discover and return all available and installed pipeline components with their installation status and categorization.

Endpoint: GET /pipeline_components

Response:

{
  "metadata": {
    "total_installed": 5,
    "total_available": 3
  },
  "category_name": [
    {
      "name": "component_name",
      "label": "Component Label",
      "version": "1.0.0",
      "status": "installed",
      "is_installed": true
    }
  ]
}

Status Codes:

  • 200 OK: Successfully retrieved components
  • 500 Internal Server Error: Failed to load components

Get Remote Components

Fetches available components from the remote repository, excluding components that already exist locally.

Endpoint: GET /pipeline_components/remote

Response:

{
  "success": true,
  "components": {
    "category_name": [
      {
        "name": "component_name",
        "label": "Component Label",
        "version": "1.0.0",
        "changelog_url": "https://...",
        "file_links": {}
      }
    ]
  }
}

Status Codes:

  • 200 OK: Successfully retrieved remote components
  • 502 Bad Gateway: Could not connect to component repository
  • 500 Internal Server Error: Unexpected error occurred

Check Component Exists

Checks if a component directory exists on the filesystem.

Endpoint: GET /pipeline_components/<component_name>/exists

Path Parameters:

  • component_name (string, required): Name of the component to check

Response:

{
  "success": true,
  "exists": true
}

Status Codes:

  • 200 OK: Check completed successfully

Component Installation

Install Component

Initiates a component installation as a background task and returns an installation ID for log streaming.

Endpoint: POST /pipeline_components/<component_name>/install

Path Parameters:

  • component_name (string, required): Name of the component to install

Request Body:

{
  "file_paths": {
    "model_file": "/path/to/model.pth",
    "config_file": "/path/to/config.yaml"
  }
}

Response:

{
  "success": true,
  "message": "Installation started.",
  "installation_id": "550e8400-e29b-41d4-a716-446655440000"
}

Status Codes:

  • 202 Accepted: Installation initiated successfully
  • 500 Internal Server Error: Failed to start installation

Stream Installation Logs

Streams real-time installation logs using Server-Sent Events (SSE).

Endpoint: GET /pipeline_components/install/stream/<installation_id>

Path Parameters:

  • installation_id (string, required): UUID of the installation job

Response Format: text/event-stream

Event Data Structure:

{
  "level": "info",
  "message": "Installing dependencies...",
  "progress": 50,
  "status": "in_progress"
}

Log Levels:

  • info: General information
  • success: Successful completion
  • error: Error occurred
  • warning: Warning message

Status Values:

  • starting: Installation is starting
  • in_progress: Installation in progress
  • completed: Installation completed successfully
  • failed: Installation failed

Download Component

Downloads a component ZIP file from a remote URL and extracts it into the pipeline_components directory.

Endpoint: POST /pipeline_components/download

Request Body:

{
  "zip_url": "https://example.com/component.zip",
  "component_name": "image_metadata_extractor"
}

Response:

{
  "success": true,
  "message": "Component 'image_metadata_extractor' downloaded successfully."
}

Status Codes:

  • 200 OK: Component downloaded successfully
  • 400 Bad Request: Missing required fields
  • 409 Conflict: Component directory already exists
  • 500 Internal Server Error: Download or extraction failed

Uninstall Component

Uninstalls a pipeline component and removes it from the database.

Endpoint: POST /pipeline_components/uninstall

Request Body:

{
  "component_name": "image_metadata_extractor"
}

Response:

{
  "success": true,
  "message": "Component image_metadata_extractor uninstalled successfully"
}

Status Codes:

  • 200 OK: Component uninstalled successfully
  • 400 Bad Request: Component name not provided
  • 500 Internal Server Error: Uninstallation failed

Component Updates

Check for Updates

Compares local installed component versions against the remote repository to identify available updates.

Endpoint: GET /pipeline_components/check_updates

Query Parameters:

  • test_mode (boolean, optional): Simulates an outdated component for testing (default: false)

Response:

{
  "success": true,
  "updates": [
    {
      "name": "image_metadata_extractor",
      "label": "Image Metadata Extractor",
      "local_version": "1.0.0",
      "remote_version": "1.1.0",
      "update_url": "https://...",
      "changelog_url": "https://...",
      "file_links": {}
    }
  ]
}

Status Codes:

  • 200 OK: Update check completed successfully
  • 502 Bad Gateway: Could not connect to component repository
  • 500 Internal Server Error: Unexpected error occurred

Update Component

Downloads and reinstalls a component to update it while preserving user-added files.

Endpoint: POST /pipeline_components/<component_name>/update

Path Parameters:

  • component_name (string, required): Name of the component to update

Request Body:

{
  "zip_url": "https://example.com/component-v2.zip"
}

Response:

{
  "success": true,
  "message": "Update process for 'image_metadata_extractor' initiated.",
  "installation_id": "550e8400-e29b-41d4-a716-446655440000"
}

Status Codes:

  • 202 Accepted: Update initiated successfully
  • 400 Bad Request: Missing zip_url
  • 404 Not Found: Component directory not found
  • 500 Internal Server Error: Update failed

Get Changelog

Proxies changelog content from a remote URL to avoid CORS issues.

Endpoint: GET /pipeline_components/changelog

Query Parameters:

  • url (string, required): URL of the changelog file

Response Format: text/markdown

Status Codes:

  • 200 OK: Changelog retrieved successfully
  • 400 Bad Request: URL parameter missing
  • 502 Bad Gateway: Could not retrieve changelog content

Component Configuration

Get Component Configuration

Retrieves component configuration including installation requirements, parameters, and stored values.

Endpoint: GET /pipeline_components/<component_name>/config

Path Parameters:

  • component_name (string, required): Name of the component

Response:

{
  "success": true,
  "component_name": "image_metadata_extractor",
  "inputs": ["image"],
  "outputs": ["metadata"],
  "parameter_groups": [
    {
      "title": "General Parameters",
      "description": "Component-specific settings.",
      "parameters": [
        {
          "name": "model_path",
          "type": "file",
          "default": "/path/to/model.pth",
          "installation_provided": true
        }
      ]
    }
  ],
  "specification": {}
}

Status Codes:

  • 200 OK: Configuration retrieved successfully
  • 404 Not Found: Component not found
  • 500 Internal Server Error: Failed to load configuration

Save Component Configuration

Saves configuration parameters for a component in the currently loaded project database.

Endpoint: POST /pipeline_components/<component_name>/config

Path Parameters:

  • component_name (string, required): Name of the component

Request Body:

{
  "parameters": {
    "model_path": "/path/to/model.pth",
    "batch_size": 32,
    "threshold": 0.75
  }
}

Response:

{
  "success": true,
  "message": "Configuration saved for image_metadata_extractor"
}

Status Codes:

  • 200 OK: Configuration saved successfully
  • 400 Bad Request: No project loaded
  • 500 Internal Server Error: Database operation failed

Component Templates

List or Save Templates

Lists available parameter templates or saves a new template for a component.

Endpoint: GET|POST /pipeline_components/<component_name>/templates

Path Parameters:

  • component_name (string, required): Name of the component

GET Request

Response:

{
  "success": true,
  "templates": ["default", "high_quality", "fast_processing"]
}

POST Request

Request Body:

{
  "template_name": "custom_template",
  "parameters": {
    "model_path": "/path/to/model.pth",
    "batch_size": 16
  }
}

Response:

{
  "success": true,
  "message": "Template 'custom_template' saved."
}

Status Codes:

  • 200 OK: Operation successful
  • 400 Bad Request: Missing required fields or invalid template name
  • 500 Internal Server Error: Operation failed

Load Template

Loads a specific parameter template for a component.

Endpoint: GET /pipeline_components/<component_name>/templates/<template_name>

Path Parameters:

  • component_name (string, required): Name of the component
  • template_name (string, required): Name of the template

Response:

{
  "success": true,
  "parameters": {
    "model_path": "/path/to/model.pth",
    "batch_size": 32,
    "threshold": 0.75
  }
}

Status Codes:

  • 200 OK: Template loaded successfully
  • 404 Not Found: Template not found
  • 500 Internal Server Error: Failed to load template

Component Maintenance

Health Check

Checks all installed components for integrity and automatically uninstalls components with missing directories or environments.

Endpoint: POST /pipeline_components/health_check

Response:

{
  "success": true,
  "message": "Cleanup complete. Automatically uninstalled 2 inconsistent component(s): component1, component2.",
  "cleaned_components": ["component1", "component2"]
}

Status Codes:

  • 200 OK: Health check completed successfully

Optimize Dependencies

Optimizes shared dependencies across all installed components.

Endpoint: POST /pipeline_components/optimize

Response:

{
  "success": true,
  "message": "Dependencies optimized successfully"
}

Status Codes:

  • 200 OK: Optimization completed successfully
  • 500 Internal Server Error: Optimization failed

Get Storage Information

Retrieves component storage usage and dependency information.

Endpoint: GET /pipeline_components/storage

Response:

{
  "total_size": "2.5 GB",
  "components": [
    {
      "name": "component1",
      "size": "500 MB",
      "dependencies": ["numpy", "torch"]
    }
  ],
  "shared_dependencies_size": "1.2 GB"
}

Status Codes:

  • 200 OK: Storage information retrieved successfully
  • 500 Internal Server Error: Failed to retrieve storage information

Error Responses

All error responses follow a consistent format:

{
  "success": false,
  "error": "Descriptive error message"
}

Common Error Codes

  • 400 Bad Request: Invalid or missing request parameters
  • 404 Not Found: Requested resource does not exist
  • 409 Conflict: Resource already exists or conflicts with existing state
  • 500 Internal Server Error: Server-side error during processing
  • 502 Bad Gateway: External service unavailable or unreachable

Data Types & Formats

Component Status

  • installed: Component is fully installed and ready to use
  • available: Component is available for download/installation

Installation Log Levels

  • info: Informational messages
  • success: Successful operation completion
  • error: Error occurred
  • warning: Warning message

File Path Format

File paths in .hdpc (Heritage Data Processor Container) format must be absolute or relative to the component directory.