Synchronizing with a Data Lake

Data lakes are essential for organizations managing large fleets across multiple brands and operational environments. This guide covers strategies for efficiently synchronizing Hiboo equipment data with your data lake infrastructure while maintaining data quality and operational efficiency.

Overview

Data lake synchronization with Hiboo leverages our unified API to streamline mixed fleet data management. While traditional fleet management faces challenges with different equipment brands and data formats, Hiboo normalizes this complexity, allowing you to focus on analytics and business value rather than data integration challenges.

How Hiboo Simplifies Mixed Fleet Data:

Unified data model: All equipment data available through consistent API endpoints regardless of brand
Standardized formats: Equipment data normalized into common units, timestamps, and structures
Reliable processing: Near real-time data processing with consistent availability patterns
Comprehensive coverage: Single API access to all your fleet data

Remaining Synchronization Considerations:

Data volume management: Large fleets still generate significant data volumes requiring efficient processing
Delayed transmissions: Some equipment may have connectivity delays that affect data freshness
Time window limits: API requests limited to 14-day windows requiring batching for historical data

Strategic Approaches:

Incremental sync: Regular collection of recent data changes
Batch processing: Scheduled bulk data collection for historical completeness
Hybrid strategy: Combining real-time incremental updates with periodic bulk validation

Initial Data Lake Population

Complete Historical Data Load

Problem: You need to populate your data lake with comprehensive historical time-series data for all equipment in your fleet for analytics, machine learning, and compliance requirements.

Solution: Use systematic bulk data collection with data kind endpoints, implementing batching strategy to handle API limitations.

Data kind history

get

/v2/fleet/equipments/{id}/{dataKind}

Returns time series data for a specific data kind for a piece of equipment.

Authorizations

x-access-tokenstringRequired

Generate a JWT when you log in. Use it in your following requests in the header as x-access-token: your-token-here

Path parameters

idinteger · int32Required

ID of the equipment

dataKindstring · enumRequired

Data kind identifier.

Example: batteryLevelPossible values:

Query parameters

sincestring · date-timeOptional

Default is beginning of yesterday UTC

untilstring · date-timeOptional

Default is end of today UTC

Responses

200

Snapshot of all devices

application/json

401

Unable to authenticate the token

application/json

get/v2/fleet/equipments/{id}/{dataKind}

GET /v2/fleet/equipments/{id}/{dataKind} HTTP/1.1
Host: api.hiboo.io
x-access-token: YOUR_API_KEY
Accept: */*

{
  "data": {
    "total": 1,
    "rows": [
      {
        "percent": 88.24,
        "date": "2026-08-11T15:25:17.347Z"
      }
    ],
    "id": 1,
    "serialNumber": "YVR001988",
    "since": "2019-11-06T00:00:00Z",
    "until": "2019-11-20T23:59:59Z"
  }
}

Implementation strategy:

1. Equipment and data prioritization:

1. Get complete equipment list from fleet endpoint
2. Identify critical data kinds required
3. Prioritize by operational importance or analytics requirements

2. Batched historical collection:

Time window: Maximum 14 days per request (API limitation)
Processing: Batch requests in 14-day chunks for complete history
Error handling: Retry failed batches with exponential backoff
Sequencing: Work backwards from recent data to oldest available

Best practices for bulk loading:

Batch time windows: Split historical data collection into 14-day chunks
Process during off-hours: Schedule bulk operations during low-activity periods
Checkpoint progress: Track completed equipment and date ranges to enable resume on failures

Batching strategy example:

For 1 year of history:
- Split into ~26 batches of 14 days each
- Process sequentially: most recent → oldest
- Track completion: asset_id + date_range + data_kind combinations
- Resume capability: restart from last successful batch

Consolidated Overview Data

Problem: You need consolidated daily summaries and multiple data types in a single request for reporting dashboards and overview analytics.

Solution: Use equipment history endpoints for multi-data-type consolidated information.

Daily history

get

/v2/fleet/equipments/{id}/history

Returns comprehensive daily history for a piece of equipment including positions, usage, and fault codes.

Authorizations

x-access-tokenstringRequired

Generate a JWT when you log in. Use it in your following requests in the header as x-access-token: your-token-here

Path parameters

idinteger · int32Required

ID of the equipment

Query parameters

sincestring · date-timeOptional

Default is beginning of yesterday UTC

untilstring · date-timeOptional

Default is end of today UTC

Responses

200

Specific equipment details with its history

application/json

401

Unable to authenticate the token

application/json

get/v2/fleet/equipments/{id}/history

GET /v2/fleet/equipments/{id}/history HTTP/1.1
Host: api.hiboo.io
x-access-token: YOUR_API_KEY
Accept: */*

{
  "data": {
    "id": 1,
    "name": "D8000-1",
    "make": "Caterpillar",
    "model": "8000",
    "serialNumber": "YVR001988",
    "positions": {
      "total": 1,
      "rows": [
        {
          "latitude": 48.858269,
          "longitude": 2.29447,
          "date": "2026-08-11T15:25:17.347Z"
        }
      ]
    },
    "faultCodes": {
      "total": 1,
      "rows": [
        {
          "levelLabel": "unknown",
          "severity": -1,
          "identifier": "text",
          "description": "text",
          "date": "2026-08-11T15:25:17.347Z"
        }
      ]
    },
    "fuelRemainings": {
      "total": 1,
      "rows": [
        {
          "percent": 88.24,
          "date": "2026-08-11T15:25:17.347Z"
        }
      ]
    },
    "engineConditions": {
      "total": 1,
      "rows": [
        {
          "running": true,
          "date": "2026-08-11T15:25:17.347Z"
        }
      ]
    },
    "dailyUsages": {
      "total": 1,
      "rows": [
        {
          "date": "2026-08-11T15:25:17.347Z",
          "operatingHours": {
            "hour": 8.3
          },
          "idleHours": {
            "hour": 1.5
          },
          "workingHours": {
            "hour": 6.8
          },
          "fuelUsed": {
            "fuelUsed": 99.7,
            "unit": "L"
          },
          "dailyDistance": {
            "odometer": 54.2,
            "unit": "km"
          },
          "cumulativeIdleHours": {
            "hour": 1064.6
          },
          "cumulativeOperatingHours": {
            "hour": 1064.6
          },
          "cumulativeFuelUsed": {
            "fuelUsed": 9520.5,
            "unit": "L"
          },
          "cumulativeDistance": {
            "odometer": 5423.9,
            "unit": "km"
          },
          "cumulativeLoadCount": {
            "count": 13569
          },
          "cumulativePayloadTotal": {
            "payload": 447839.9,
            "payloadUnit": "tonnes"
          },
          "cumulativeActiveRegenerationHours": {
            "hour": 10.92
          },
          "cumulativePowerTakeOffHours": {
            "hour": 31.21
          },
          "peakDailySpeed": {
            "speed": 14.91,
            "speedUnit": "km/h"
          },
          "averageDailyEngineLoadFactor": {
            "percent": 35.9
          },
          "DEFRemainingRatio": {
            "percent": 62,
            "tankCapacity": 41,
            "tankCapacityUnit": "L"
          },
          "co2": {
            "weight": 123,
            "unit": "kg"
          }
        }
      ]
    }
  }
}

What this endpoint provides:

Daily operational summaries: Engine hours, fuel consumption, distances
Multiple data types: Positions, fault codes, fuel levels, engine conditions in one response
Consolidated format: Pre-aggregated data suitable for overview reporting

Use this approach when:

Dashboard population: Creating fleet overview dashboards
Reporting summaries: Generating daily/weekly operational reports
Multi-data-type collection: Need several data types simultaneously

Note: For complete detailed time-series data, use the /{dataKind} endpoint described above.

Incremental Synchronization

Real-Time Data Sync

Problem: You need to keep your data lake current with near real-time equipment data for operational dashboards and immediate decision-making.

Solution: Implement incremental sync using snapshot endpoints combined with targeted historical data collection.

Fleet

get

/v2/fleet/equipments/snapshot

Returns the last information available for all data kinds for the fleet. This endpoint is paginated.

Authorizations

x-access-tokenstringRequired

Generate a JWT when you log in. Use it in your following requests in the header as x-access-token: your-token-here

Query parameters

limitinteger · int32 · max: 50Optional

The numbers of items to return

Default: 20

offsetinteger · int32Optional

The number of items to skip before starting to collect the result set

Default: 0

searchstringOptional

Optional - if provided it will return the equipment with a fleet number or serialNumber matching the parameter

filterByInProgressSharingintegerOptional

This parameter allows filtering the equipments based on whether the equipment is currently involved in an in-progress sharing process or not.

Example: {"value":0,"summary":"Filter by not in progress sharing"}

Responses

200

Snapshot of all devices

application/json

401

Unable to authenticate the token

application/json

get/v2/fleet/equipments/snapshot

GET /v2/fleet/equipments/snapshot HTTP/1.1
Host: api.hiboo.io
x-access-token: YOUR_API_KEY
Accept: */*

{
  "data": {
    "total": 1,
    "rows": [
      {
        "id": 1,
        "name": "D8000-1",
        "make": "Caterpillar",
        "model": "8000",
        "serialNumber": "YVR001988",
        "cumulativeDistance": {
          "date": "2019-11-06T00:00:00Z",
          "odometer": 54.2
        },
        "cumulativeFuelUsed": {
          "date": "2019-11-06T00:00:00Z",
          "fuelUsed": 9520.5
        },
        "fuelRemaining": {
          "date": "2019-11-06T00:00:00Z",
          "percent": 65.5
        },
        "cumulativeOperatingHours": {
          "date": "2019-11-06T00:00:00Z",
          "hour": 1.5,
          "usageHours": 1.5
        },
        "lastPosition": {
          "date": "2019-11-06T00:00:00Z",
          "latitude": 48.858269,
          "longitude": 2.29447
        },
        "cumulativeLoadCount": {
          "count": 1,
          "date": "2019-11-06T00:00:00Z"
        },
        "cumulativePayloadTotal": {
          "date": "2019-11-06T00:00:00Z",
          "payload": 1
        },
        "cumulativeActiveRegenerationHours": {
          "date": "2019-11-06T00:00:00Z",
          "hours": 15
        },
        "cumulativePowerTakeOffHours": {
          "date": "2019-11-06T00:00:00Z",
          "hours": 12
        },
        "peakDailySpeed": {
          "date": "2019-11-06T00:00:00Z",
          "speed": 100.5
        },
        "averageDailyEngineLoadFactor": {
          "date": "2026-08-11T15:25:17.347Z",
          "percent": 88.24
        },
        "DEFRemainingRatio": {
          "date": "2026-08-11T15:25:17.347Z",
          "percent": 58.24
        },
        "cumulativeIdleHours": {
          "date": "2026-08-11T15:25:17.347Z",
          "hours": 12
        }
      }
    ]
  }
}

Sync schedule:

High-frequency sync (every 15-30 minutes):

Target: Current operational status
Data: Fleet snapshots, individual equipment snapshots
Purpose: Real-time dashboards, operational monitoring
Storage: Current state tables, operational data store

Medium-frequency sync (every 2-4 hours):

Target: Recent historical data
Data: Last 24-48 hours of detailed data
Purpose: Recent trend analysis, operational reporting
Storage: Time-series tables, recent activity store

Delayed Data Recovery

Problem: Equipment data transmission delays mean your data lake may be missing recent data that arrives later than expected.

Solution: Implement systematic delayed data recovery using overlapping time windows.

Recovery strategy:

Weekly validation sync:

Time window: Last 10 days (comprehensive delayed data capture)
Schedule: Weekly on weekend nights
Process: Full data validation and gap filling
Focus: Complete data integrity and weekend transmission delays

Implementation pattern:

1. Query data lake for existing data in time window
2. Query Hiboo API for same time window
3. Identify gaps and changes
4. Update data lake with missing/updated records
5. Log data quality metrics and delay patterns

PreviousSetting up efficient polling strategies NextExporting cumulative data to your systems

Last updated 1 year ago

Was this helpful?