The AIS Data Model

The current “data model” used in the project is a combination of the AIS protocol, some database schema’s that were convenient and whatever made sense at the time. There were more and more edge cases forced into the code as time went on. This became apparent to me when trying to explain the whole project to someone else, I kept having to try doing a brain dump and providing history to explain concepts. That’s not ideal..

But it works.

So if we were to take a more deliberate approach, and implement a designed data model, what would that entail? A database redesign, of course, and refactoring the ETL pipeline (but that should have been done years ago anyway). It would however give us a bunch of extra links, a clear concept of how things fit together, and might point out some low hanging fruit that I’ve missed until now. The current data model (let’s be generous and pretend it’s a data model) ingests AIS Position and Voyage Reports and links them together by MMSI. It does some aggregations (many positions become trajectories, and many voyage reports become a “last observation carried forward” table.) and tries to reconcile many-to-one issues with some rules:

• Sometimes multiple vessels have the same MMSI; split by distance
• Sometimes multiple AIS transmitters belong to the same vessel; smash the messages together

This just feels a little bit swampy. There’s friction, there’re edge cases, there’s history.

From a philosophical point of view, what is an AIS message? I’d say it’s a noisy snapshot that provides a limited window on the identity and movement of an AIS equipped “thing”. Most of the time that thing is a ship, but it could be a lighthouse, buoy, aircraft, net marker, vessel towed behind a car, phantom, spoofing artifact etc etc. Let’s say that for our case it’s a ship, what kind of ship would it be? The AIS protocol gives us some choices to make but recent research has pretty much shown that the protocol options aren’t good enough. Give me a choice of 100 options and I’ll run into the 101’st. I might be biased having spent time with real data scientists (scientists studying data vs scientists studying something represented by data.) but it seems like using Types and controlled vocabulary should help out here.

Vessels have a “type” and “cargo”/“tanker”/“fishing” are types of vessel. This concept is better explained using linked data, semantic triples, and RDF concepts. Using triples; when I run into the 101’st edge case I can make several new statements; “This vessel is a net marker.” “A net marker is a type of AIS carrying platform.” “A net-marker is associated to a fishing vessel” and one day a sensible software tool could make use of it. Until then it would at least pass any typing checks and allow me to build a classifier that could identify net markers.

What else is an AIS message? Describing the movement of a vessel is a big part of it; and movement can lead to trajectories, events (vessel entered MPA, or vessel stopped moving), interactions (transshipment between vessels, vessels reaching ports) and eventually big ol’ aggregates (number of hours vessels have spent fishing in a geographic region). Looking at interactions implies a bunch of objects that are NOT described by AIS; ports, EEZ’s, MPA’s, (and the less well defined) anchorages, shipping lanes, anchor scars… the list is large.

Let’s start mapping this all out, I’m not going to focus on naming now since it’s been proven to me that I’m no good at it. We’ll have several data entities:

• Ships/Vessels/Platforms/Things: An artificial platform that performs some function in (or near) the ocean or waterway. This will be a ship most of the time, and carry AIS, but doesn’t have to be.
• Place/location/zone: A place, that is be defined using geometry, that is distinct from other places. Is a point of interest for vessels moving around; port, anchorage, fishing area, traffic lane, marine protected area. Can be the edge-point of a vessel voyage + events graph: “ship left port A, ship travelled to fishing zone B, ship spent 15 hours in state “fishing” in zone B, etc”
• Events: Something that happened that changed the legal, physical, or logical state of a Ship (or place?). It must have a timestamp, and probably associated location, and would be of interest to human operators: Ship X moved into a marine protected area, Ship stopped travelling, ship changed name from X to Y.
• Position Reports: AIS position reports! Location, speed, heading etc.
• Voyage Reports: AIS ID messages! ETA, names, call-signs etc. There doesn’t seem to be a need to actually record most of these. If anything these can be used to generate events and otherwise update the Ships entity.
• Trajectories/Voyages/Position Aggregates: A collection of Position Reports that are grouped by some kind of logical category: positions per day, trajectory between ports. It seems that these should be split by events, even if the event is as simple as “lost contact with vessel”. They could also be linked to prev and next trajectories.

A quick example:

• A ship with MMSI 12345 is sending many AIS position and voyage report messages today
  • Ship Entity is created with Name, MMSI etc extracted from Voyage and Position reports
  • A bunch of new Positions is created
• The AIS reports indicate it has left Port A and may be travelling to Port B, in a different country
  • Port A and B are already defined in the data store, probably from the World Port Index
  • There is a bunch of geometry between A and B; 12 nautical mile lines, and EEZ.
  • There is also spotty AIS coverage between the two countries
• A new Trajectory Aggregate is calculated every hour, and compared to Places to determine whether a new Event has taken place:
  • A trajectory of type “Hourly Update” is calculated
  • Event 1: Vessel 12345 has left Port A at 08:00
  • Event 2: Vessel 12345 has left Country X 12NM zone at 10:30
  • Event 3: No AIS has been received from Vessel 12345 since 11:30
  • Event 4: AIS from Vessel has been received from Vessel 12345 at 22:00
  • Event 5: Vessel 12345 has entered Port B at 23:20
• A voyage has been detected as being concluded:
  • A new Voyage from Port A to Port B by Vessel 12345 is recorded
  • This Voyage is made up of two trajectories; 08:00 to 11:30 and 22:00 to 23:20

This is a first draft, and serves the purpose of sparking discussion with the VLIZ data science folk, and to get back into writing blog posts on this page…