Server Installation Script Reference¶

The Heritage Data Processor Server Installation Script (install_server.sh) installs HDP server components in editable/development mode using pyproject.toml for package management.

Overview¶

This script provides automated installation of HDP server components with support for development dependencies, virtual environment management, and editable package installation for active development workflows. It enables the usage of the commands hdp and hdp-server within the virtual environment.

Usage¶

Basic Usage¶

./install_server.sh

Installs HDP server components with default settings: creates .venv virtual environment with Python 3.11 and installs packages in editable mode.

Command-Line Options¶

./install_server.sh [OPTIONS]

Options:

--force: Force reinstall even if virtual environment already exists. Removes and recreates .venv
--dev: Install with development dependencies (testing tools, linters, etc.)
--python VERSION: Python version to use. Default: 3.11
--venv PATH: Virtual environment path. Default: .venv
--help: Display usage information and exit

Requirements¶

Required Tools¶

uv: Python package installer and virtual environment manager (https://docs.astral.sh/uv/)
Python 3.11+: Required for project compatibility

Required Files¶

pyproject.toml: Must exist in project root directory

System Resources¶

Disk Space: Sufficient space for packages and dependencies (typically 200-500 MB)
Network: Internet connectivity for package downloads

Installation Steps¶

Step 1: System Validation¶

Validates system prerequisites before beginning installation.

Checks Performed:

Project Structure: Verifies pyproject.toml exists in current directory
uv Availability: Confirms uv is installed and accessible
Python Version: Checks for specified Python version (defaults to 3.11)
Project Name: Extracts project name from pyproject.toml if possible

Exit Conditions:

If pyproject.toml is missing or uv is not installed, the script exits with installation instructions.

Example Output:

[INFO] Step 1: System Validation
✅ Found pyproject.toml
[DEBUG] uv: uv 0.1.20
✅ uv is installed
[INFO] Checking for Python 3.11...
[DEBUG] Found: Python 3.11.5
[INFO] Project: heritage-data-processor
✅ Pre-flight checks completed

Step 2: Virtual Environment Setup¶

Creates or verifies Python virtual environment.

New Installation:

Creates virtual environment using:

uv venv --python=3.11

Existing Environment:

Standard Mode: Uses existing .venv if valid, skips recreation
Force Mode (--force): Removes and recreates .venv regardless

Validation:

After creation, verifies: - Activation script exists at .venv/bin/activate - Python version matches requested version

Force Reinstall Trigger:

When --force is specified:

[INFO] Force reinstall requested, removing existing virtual environment...
[INFO] Creating virtual environment with Python 3.11...

Example Output:

[INFO] Step 2: Virtual Environment Setup
[INFO] Virtual environment already exists: .venv
✅ Using existing virtual environment
[DEBUG] Virtual environment Python: Python 3.11.5

Step 3: Package Installation¶

Installs HDP packages in editable mode from pyproject.toml.

Installation Command:

Standard Mode:

uv pip install --python .venv -e .

Development Mode (--dev):

uv pip install --python .venv -e .[dev]

Editable Mode Benefits:

Source code changes are immediately reflected
No reinstallation needed after modifications
Ideal for active development

Development Dependencies:

When --dev is specified, the script checks for optional dependencies: - [project.optional-dependencies] section in pyproject.toml - [tool.uv.dev-dependencies] section

If no dev dependencies found, warns user but continues installation.

Progress Indication:

[INFO] Installing packages in editable mode from pyproject.toml...
[INFO] Running: uv pip install --python .venv -e .
[INFO] This may take a few minutes depending on package sizes...

Error Handling:

Provides detailed diagnostics for common failures:

🔴 Failed to install packages.

Common causes:
 - Package version conflicts
 - Network timeout
 - Missing system dependencies (e.g., gcc, python3-dev)
 - Corrupted package cache

Check the log for details: install_hdp_server_20251021_153000.log

Try manually:
   source .venv/bin/activate
   uv pip install -e . --verbose

Step 4: Installation Verification¶

Verifies successful installation and displays package information.

Verification Process:

Activate Environment: Sources .venv/bin/activate
Count Packages: Lists all installed packages
Identify Main Packages: Searches for common package names:
heritage-data-processor
hdp
hdp-server
heritage_data_processor
Check Editable Mode: Verifies packages are installed in editable mode
Display Versions: Shows version numbers for identified packages

Example Output:

[INFO] Step 4: Verifying Installation
[INFO] Activating virtual environment for verification...
[INFO] Checking installed packages...
[DEBUG] Total installed packages: 47
✅ Verified packages: hdp hdp-server
[DEBUG] hdp version: 0.1.0
[DEBUG] hdp-server version: 0.1.0
✅ Package installed in editable mode (development mode)
[DEBUG] Editable packages:
  hdp                      0.1.0  /path/to/project
  hdp-server               0.1.0  /path/to/project
✅ Installation verification completed

Step 5: Post-Installation Setup¶

Performs final configuration tasks.

Installation Marker File:

Creates .hdp_server_installed containing:

INSTALLED_DATE=2025-10-21 15:30:00
VENV_PATH=.venv
PYTHON_VERSION=3.11
DEV_MODE=false

Purpose: Tracks installation for troubleshooting and status verification.

Additional Checks:

Detects start_hdp.sh if present
Identifies config/ directory if exists

Output & Logging¶

Log File¶

All operations are logged to timestamped file:

Format: install_hdp_server_YYYYMMDD_HHMMSS.log

Location: Current working directory

Content: - All commands executed - Command output - Diagnostic information - Error messages with stack traces

Console Output¶

Color-Coded Messages:

🔴 Red: Critical errors
⚠️ Yellow: Warnings
✅ Green: Success messages
🔵 Blue: Debug information
🔷 Cyan: Section headers and commands

Log Levels:

ERROR: Critical failures
WARN: Warnings that do not stop installation
INFO: General information
DEBUG: Detailed diagnostic data

Error Handling¶

Common Error Scenarios¶

pyproject.toml Not Found:

🔴 pyproject.toml not found in current directory.
This script must be run from the project root directory.

Current directory: /home/user/wrong-location

uv Not Installed:

🔴 uv not found.

Install uv:
   curl -LsSf https://astral.sh/uv/install.sh | sh

After installation, restart your terminal and try again.

Python Version Not Available:

🔴 Failed to create virtual environment.

Common causes:
 - Python 3.11 not installed
 - Insufficient permissions
 - Corrupted uv installation

Install Python 3.11:
 - Ubuntu/Debian: sudo apt install python3.11
 - macOS: brew install python@3.11
 - Windows: Download from python.org

Package Installation Failure:

🔴 Failed to install packages.

Common causes:
 - Package version conflicts
 - Network timeout
 - Missing system dependencies (e.g., gcc, python3-dev)
 - Corrupted package cache

Check the log for details: install_hdp_server_20251021_153000.log

Try manually:
   source .venv/bin/activate
   uv pip install -e . --verbose

Installation Summary¶

Upon successful completion, displays comprehensive summary with next steps:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ HDP Server Installation Complete!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Installation Summary:
   Virtual Environment: .venv
   Python Version: 3.11
   Install Mode: Editable/Development
   Duration: 87s
   Log File: install_hdp_server_20251021_153000.log

Next Steps:

   1. Activate the virtual environment:
      source .venv/bin/activate

   2. Start the HDP server (runs in foreground):
      hdp-server

      Optional server arguments:
      hdp-server --port 8080                    # Custom port
      hdp-server --config custom-config.yaml    # Custom config
      hdp-server --alpha-features               # Enable experimental features

   3. Use HDP CLI commands (in a new terminal):
      source .venv/bin/activate
      hdp --help                                # View all commands
      hdp component-list                        # List available components
      hdp create --help                         # Create a new project

💡 Quick Start Examples:

   Create a new project:
   hdp create --hdpc-path myproject.hdpc \
     --project-name "My Heritage Project" \
     --short-code "MHP2025" \
     --modality "3D Model" \
     --input-dir /data/models \
     --output-dir /data/output

   Upload and create Zenodo drafts:
   hdp upload --hdpc myproject.hdpc \
     --input-dir /data/files \
     --sandbox

   Inspect project status:
   hdp inspect --hdpc-path myproject.hdpc

📚 Additional Resources:

   - CLI automatically manages the server (starts when needed)
   - Use --no-auto-server flag to manage server separately
   - Server runs on http://127.0.0.1:5001 by default
   - Changes to source code are immediately reflected (editable install)

🔧 Development Tips:

   - No reinstall needed after code changes (editable mode)
   - View installed packages: uv pip list
   - View editable packages: uv pip list --editable
   - For full CLI documentation, see CLI_Documentation.md

Usage Examples¶

Example 1: Standard Installation¶

./install_server.sh

Installs with default settings: Python 3.11, .venv path, no dev dependencies.

Example 2: Development Installation¶

./install_server.sh --dev

Installs with development dependencies for testing and development tools.

Example 3: Force Reinstall¶

./install_server.sh --force

Removes existing virtual environment and performs fresh installation.

Example 4: Custom Python Version¶

./install_server.sh --python 3.12

Uses Python 3.12 instead of default 3.11.

Example 5: Custom Virtual Environment Path¶

./install_server.sh --venv venv

Creates virtual environment at venv/ instead of .venv/.

Example 6: Complete Custom Setup¶

./install_server.sh --dev --force --python 3.12 --venv .venv-dev

Performs force reinstall with Python 3.12, development dependencies, and custom venv path.

Post-Installation Workflow¶

Two-Terminal Setup¶

Terminal 1 - Server:

source .venv/bin/activate
hdp-server

Server runs in foreground, displays logs in real-time.

Terminal 2 - CLI Operations:

source .venv/bin/activate
hdp component-list
hdp create --hdpc-path myproject.hdpc ...
hdp upload --hdpc myproject.hdpc ...

Single-Terminal Setup (Automatic Server)¶

source .venv/bin/activate
hdp upload --hdpc myproject.hdpc --input-dir /data/files

CLI automatically starts and stops server as needed (default behavior).

Troubleshooting¶

Virtual Environment Corrupted¶

Symptom: Activation fails or Python version incorrect

Solution:

./install_server.sh --force

Missing Development Dependencies¶

Symptom: Dev tools not available after installation

Solution:

./install_server.sh --dev --force

Package Import Errors¶

Symptom: Cannot import installed packages

Solution:

Verify editable installation:

source .venv/bin/activate
uv pip list --editable

Should show packages installed from local directory.

Permission Errors¶

Symptom: Cannot create virtual environment or install packages

Solution:

Check directory permissions:

ls -la .
# Ensure write permissions for current user
chmod u+w .

Development Workflow¶

Making Code Changes¶

Edit Source Code: Modify Python files in server_app/ or other directories
No Reinstall Needed: Changes are immediately available (editable mode)
Restart Server: If server is running, restart to load changes:
Press Ctrl+C in server terminal
Run hdp-server again

Testing Changes¶

source .venv/bin/activate
python -c "import server_app; print(server_app.__version__)"
hdp component-list  # Test CLI commands

Updating Dependencies¶

If pyproject.toml changes:

source .venv/bin/activate
uv pip install -e .  # Reinstall with new dependencies

Or use force reinstall:

./install_server.sh --force

Comparison with Other Installation Scripts¶

vs. install_hdp.sh¶

install_hdp.sh: Clones repository, installs via requirements.txt, includes Node.js dependencies
install_server.sh: Installs from existing project directory using pyproject.toml, Python-only

vs. start_hdp.sh¶

start_hdp.sh: Validates existing installation and starts application
install_server.sh: Performs initial installation, does not start services

Use Case¶

Use install_server.sh when: - Using the HDP Server and CLI is required for your use case - Developing HDP components - Installing from local source code - Need editable package installation - Working with pyproject.toml based projects