Database Inserter

The goal of this container is to insert decoded AIS messages, pulled from the message broker, into a database. Pretty simple stuff. It handles creating the connection to the DB and doing bulk inserts periodically. This is more efficient than inserting every message individually.

Input Data

The service connects to RabbitMQ, creates a message queue that accepts messages that contain the relavant routing key. These messages are decoded AIS messages that could fall into one of the many AIS message types.

The DB Sink inserts the messages into a specific DB table depending on AIS message type (the below example is of a type 24 AIS message). Some messages contain position information that goes into the pos_report table while other contain voyage report information, going into the voy_reports table. Some messages contain both sets, or a subset, of the data and are inserted into multiple tables.

{"ais": "!AIVDM,1,1,,B,H7OfI?PPu9<rM8uDp@000000000,2*41\r", 
"header": null, 
"server_time": "2022-08-31T11:35:05.326355", 
"event_time": "2022-08-31T11:35:05.326357", 
"routing_key": "ais.aishub.all", 
"multiline": false, 
"parsed_msg": {"header": null, 
                "footer": null, 
                "event_time": null, 
                "talker": "!AIVDM", 
                "frag_count": "1", 
                "frag_num": "1", 
                "seq_id": "", 
                "radio_chan": "B", 
                "payload": "H7OfI?PPu9<rM8uDp@000000000", 
                "padding": "2", 
                "checksum": "41"}, 
"decoded_msg": {"id": 24, 
                "repeat_indicator": 0, 
                "mmsi": 503028030, 
                "part_num": 0, 
                "name": "HORSN`ROUND"}, 
"source_key": "encoded_ais.aishub.all"}

Config

The service is configured using an environment variable file. An example can be found in ./config/sample.env. This config file is loaded into the docker container on run (generally with docker-compose).

The config file is split up into several different catagories. Below is a short explanation of them.

#----------------------
#Project
#----------------------
PROJECT_NAME=ais            - The name of the project, this is prepended to all the containers being run
UID=1000                    - UID and GID of the user that will end up owning files created by the containers
GID=1000

#----------------------
# Database with Postgis and TimescaleDB
#----------------------
TIMESCALE_TAG=1.7.4-pg12    
DB_HOST=server.com          - DB Host and ports
DB_EXT_PORT=5432
DB_INT_PORT=5432
# these should be docker secrets...
POSTGRES_USER=inserter      - DB User details
POSTGRES_DB=vessels
POSTGRES_PASSWORD=inserter_super_password
 
#----------------------
# RabbitMQ
#----------------------  
RABBIT_MSG_PORT=5672        - RabbitMQ connection details
RABBIT_HOST=rabbit.com
# these should be docker secrets...
RABBITMQ_DEFAULT_USER=rory
RABBITMQ_DEFAULT_PASS=rory_pw
RABBIT_EXCHANGE=ais_proc_chain
 
#----------------------
# DB Inserter
#----------------------
INSERTER_USER=inserter
INSERTER_PW=inserter_super_password
INSERT_PERIOD=5 
INSERT_SIZE=500
BIND_TO_KEYS=["ais.imis.sat", "ais.samsa.sat", "ais.tnpa.coastal"]
INSERT_QUEUE=ais_proc_chain_sink
QUEUE_MAX_LENGTH=100000
ON_ERROR_DROP_MSGS=True

Deployment

The system is generally deployed using docker-compose. There is a compose file stored in the repo that shows how to start the container, mount volumes, use config files in the .env file. The steps to get this running on your system is:

Clone/Pull the repo to the target machine
Copy the ./config/sample.env to .env and edit it to reflect your environment
run “docker-compose up –build -d”

That will get the service running on your machine but will be pretty useless without any of the other services. There is work being done on a generic deployment that can be found in the deployment project in the OpenAIS namespace.