Automatic Identification System (AIS) is a maritime technology where ships broadcast real-time information to other ships and coastal authorities. AIS was introduced already in the late 1990s but since 2002, the International Maritime Organisation (IMO) has set AIS as mandatory for vessels over 300 gross tonnage (GT) on international voyages, cargo ships over 500 GT, and all passenger ships, as part of the SOLAS (Safety of Life At Sea) agreement which required these vessels to fit a Class A AIS transceiver . Later in 2006, simpler and cheaper Class B AIS transceivers were introduced that can be used on recreational boats. AIS operates by broadcasting real-time data as short binary messages using the 162 MHz VHF frequency reserved for the purpose. The messages do not target specific recipients, but the transceivers process messages from all other transceivers within reach. Terrestrial receivers are set up by authorities or by anyone interested in monitoring the ship traffic. As the range of AIS signal is limited to <100 km under normal propagation conditions, monitoring the marine traffic further away from coast relies on AIS satellites owned by governmental institutions or commercial enterprises.

This study presents the experimental AIS set-up at the Utö Atmospheric and Marine Research Station, located on the island of Utö, in the Archipelago Sea of the Baltic Sea (59°46^′50 N, 21°22^′23 E). The research station has a long history of atmospheric and marine observations dating back to 1881 . The region has a lot of potential for AIS based ducting research and monitoring with many busy sea routes near the island and in the study area. The study area is shown in Fig. a. The complex archipelago is shown in panel c based on the European digital elevation model (v. 1.1) from the Copernicus Land Monitoring Service .

In July 2023, two AIS receiver systems were installed at Utö. Each AIS receiver system consists of VHF and GPS antennae, receiver, and data logger (Fig. ). The VHF antennae are Comrod AV7 antennae with vertical polarization and an antenna gain of 2 dBi. The installation heights were set by technical considerations such as location of other close-by radio transmitters and protection against the sea spray. Both antenna cables were set at the same length of 120 m, which introduces approximately 7 dB loss. Although this reduces the sensitivity, it ensures that the set-ups are comparable. The mast set-up is shown in Fig. b.

Kongsberg AIS RX610 receivers are used for receiving all incoming AIS messages from Channels 1 (161.975 MHz) and 2 (162.025 MHz). The Received Signal Strength Indicator (RSSI) of each message is logged, with the sensitivity of the receiver being -115 dBm which is more sensitive than the typical -109 dBm . The incoming messages are pre-processed and saved into hourly data files. The messages are decoded in near-real-time using the freely available Python library Pyais for binary AIS message decoding (https://pypi.org/project/pyais/, last access: 8 May 2024). After decoding, further preprocessing includes discarding messages that have virtual_aid parameter that is non-zero or lack location information. The virtual aid parameter is used to exclude virtual Aid-to-Navigation (AtoN) messages that are simulated by nearby AIS stations rather than broadcasted by the real AtoN (e.g. a buoy). The data used for analysis includes time, message type, MMSI, RSSI and location. Based on the location, transmitter distance and the azimuth from the receiver are calculated for each message.

The global AIS data used in this study was provided by ORBCOMM Ltd. The global AIS data is a combination of terrestrial and satellite AIS data. ORBCOMM has a commercial satellite network with 18 AIS-enabled satellites. This results in up to 135 satellite passes and overhead coverage up to 90 %. In total 8.9 billion messages were received in 2023. The ORBCOMM global AIS data is independent from the AIS data used in this study and therefore was used as a background “truth” to establish the areal coverage of the experimental AIS set-up over two months, September–October 2023. As the global AIS combines satellite and terrestrial AIS data, it is used as the background truth in the analysis, assuming it shows all vessels present in the study area. However, the global dataset can also be affected by anomalous propagation which can introduce error in the analysis.

The line-of-sight distance (R) is the maximum distance at which a radio signal can be transmitted and received due to the curvature of the Earth under standard refractivity conditions. It depends on the heights of the receiving (HR) and transmitting antennae (HT) and can be calculated: 1R=2ak×(HR+HT), where a is the earth radius and k is the earth factor that is 4/3 for standard atmosphere . The line-of-sight distances for both 7 and 30 m antenna are shown in Fig. .

The role of diffraction is greater at lower frequencies. Hence, the maximum range of terrestrial AIS is governed by line-of-sight and diffraction propagation . The smooth earth diffraction propagation loss at 162 MHz can be described: 2Ldiff=LFS+F(X)+G(YT)+G(YR),LFS=20log⁡(D)+20log⁡(f)+20log⁡4πc,F(X)=11+10log⁡(X)−17.6×X,X=0.029D,G(YT,R)=20log⁡(YT,R+0.1×YT,R3),YT,R=0.014HT,R, where LFS is the free space propagation loss (dB), f is the frequency (Hz), c is speed of light (m s⁻¹), F(X) is the distance factor (dB), D is the distance separator, G(YT,R) are the height gain factors (dB) and H is the transmitter and receiver antenna heights (m) . Received power (dBm) can be estimated: 3PRec=EIRP-Ldiff+GR+GCorr-Lmisc, where EIRP is the ship-borne AIS equivalent isotropic radiated power (dBm) typically 41 dBm for Class A vessels, GR is the receiver antenna gain (dBi), GCorr is the receiver correlation gain (dB) (assumed as 5 dB here) and Lmisc is the miscellaneous cable losses (assumed to be 7 dB here) . The estimated received power (dBm) as a function of distance for the two Utö antennae can be seen in Fig. . The maximum AIS range, based on the receiver sensitivity, can be estimated to be 42–67 km for the 7 m antenna and 64–89 km for the 30 m antenna. The range increases with the height of the transmitting antenna.

Information about AIS antenna heights is not readily available and varies between ships as there are no IMO regulations for AIS antenna height or placement on the ship, although it is typically in the mast superstructure above the bridge. For the Baltic ferries this can be up to 50 m. Thus in Utö AIS data, a ship exceeding a certain distance suggested by the range formulas could be either a tall ferry or a small boat observed due to anomalous propagation conditions. AIS data from individual ships with known transceiver specifications could be used to validate the range formulas and study how they can be used to detect anomalous propagation and the associated atmospheric conditions. However, the objective of the present study is to use the AIS data en masse to identify propagation conditions and their seasonal and diurnal variations in the study area extending to nine Baltic basins. Hence, empirical and statistical methods are experimented in order to establish over-the-horizon (OH) propagation criteria and to identify periods of OH observations. The selected basic descriptor for each antenna is the 95th percentile of maximum distance, that is, the radius of a circular boundary such that for 95 % of the ships the maximum distance detected during a time period is within the boundary. The 95th percentile was chosen as it was more descriptive of OH observations than the median and less sensitive to individual ships than the 99th percentile. Furthermore, the AIS data was gridded into 0.25°×0.25° grid and hourly visibility was calculated for each grid and visualised as percentages. This allows for identifying the horizon based on the global AIS product. Hourly visibility is the percentage of hours during which at least one message was received from the grid, and therefore 100 % visibility is reached if at least one signal was received every hour during the 1-year period, August 2023–July 2024.

In this study, profiles of the modified refractivity (M-profiles) were calculated based on measurements of air temperature and relative humidity at heights 4, 7, 12, 22, 32 and 59 m a.m.s.l. similarly to . From the M-profiles, vertical gradients of M (dM/dh) were calculated. The vertical layer in the M-profile where dM/dh<0 is the trapping layer of the duct. The top height of the trapping layer is the duct height. Duct strength (intensity) (ΔM) is defined as the absolute change in M in M-Units (MU) in the trapping layer.

The AIS receivers record a large number of messages per day (Fig. ). On average, after preprocessing, the receiver at 7 m received 198 116 valid messages per day and the 30 m receiver 477 462 messages per day over the study period. The number of received and preprocessed messages are strongly correlated (r=0.98, p<0.05). During the preprocess, on average 34 % of the daily received messages are discarded from the 7 m AIS antenna data and 29 % of the 30 m antenna data. The number of discarded messages increases during winter and peaks in early spring, where nearly 50 % is discarded during preprocess. This is due to the increase in virtual AtoNs over winter due to sea ice in the Baltic Sea (Fig. ).

The number of messages is relatively stable from October 2023 to February 2024 and unstable from August to September 2023 and March to July 2024 (Fig. ). The number of valid received messages depends on multiple factors; characteristics of the receiving antenna (e.g. height and sensitivity), characteristics of the transmitting antenna (e.g. height and power), any filtering performed in post-processing, the amount of ships within reach, surrounding topography (shadowing) and atmospheric conditions. Majority of these aforementioned factors are stable and any variance in the number of messages is caused by the number of ships within reach, potential shadowing of antennae by other ships and atmospheric propagation conditions. The differences in the number of received messages between the two antennae are most likely due to the lower antenna having a lower range and it being more susceptible to shadowing.

The directional distribution of Class A and Class B position reports, their RSSI and distance from Utö, in a circular plot (wind rose) divided into eight segments (cardinal and ordinal directions) for two months (September and October 2023) can be seen in Fig. . The percentages correspond to the proportion of the messages that are received from each direction during a month long period. It appears that for October when the daily number of messages was mostly stable, most messages are received from east and northwest, within 0–50 km for the 7 m antenna and 0–100 km for the 30 m antenna. As expected, the share of messages received further away is greater for the 30 m antenna.

Directly to the east is the guest port of Utö, where most of the strongest messages (RSSI >-71) are transmitted from. The guest port is especially strongly represented in the 7 m antenna data, as 30 % of the messages are received from within 1 km to the east in October. For September, when the daily number of messages was unstable with some peaks, the share of messages received from south, and from within the sea sector of the mast (roughly from northwest to southeast) is increased. This decreases the relative share of east in the charts. The share of weaker messages received further away also increases beyond 100 km. This indicates that while the number of messages is increased, some of the messages are also transmitted from further away. The roses for all months during the study period (August 2023–July 2024) can be seen in Figs.  and (Appendix ).

In order to identify anomalies in the AIS range and study their occurrence, the hourly maximum distance of each vessel from Utö was calculated based on Class A and Class B position reports. Based on these maximum distances, daily 5th, 50th (median), and 95th percentiles of distance were then computed (Fig. ). The nearest 1 km from Utö is disproportionally represented in the data (up to 30 %) (Fig. ) and hence excluded before calculating the maximum distances.

The 95th percentile is more reactive to changes in the AIS range than the median. The time series suggest the presence of a horizon within which the messages are received under standard propagation conditions (within horizon, WH), and periods of anomalous conditions with over-the-horizon (OH) observations. However, the distance to the horizon cannot be expected to be sharply defined as it varies with the characteristics of the shipboard transmitters (e.g. height of the transmitting antenna).

Instead, the horizon was studied in terms of a statistical distribution model for the 95th percentile of hourly maximum distance understood as a random variable exp(X). The data north of Utö was excluded to limit the effects of the archipelago which obstructs signal propagation and restrains the traffic to few fixed routes. This leaves the spatially more scattered traffic in the open sea for the analysis. The model is defined in terms of the logarithm X of the percentile distance. The histogram for X was bimodal for both antenna heights, indicating an overlapping superposition X=X1+X2 resulting from standard (X1) and anomalous (X2) propagation. The superposition was resolved by fitting to both components the generalised normal distribution (Subbotin distribution): 4f(x)=Kexp⁡-|x-μ2σ|s,K=s22σΓ(1/s). The fitting was done in terms of half-distributions from zero to mode for X1 and from mode to infinity for X2 as described in where a similar model was used to separate WH and OH components of X-band radar clutter. The result is shown in Fig. . The component distributions for the 95th percentile distance, exp(X1) and exp(X2), are thus of generalised lognormal type . The general argument for the applicability of this distribution family is that the decrease of signal power is expected to be a multiplicative result of several factors of different origin.

The superposition weight for X2 is 0.49 and 0.68 for 7 and 30 m antenna respectively. The parameter s controls the shape of the Subbotin distribution and for (X1,X2) has values (1.19,1.68) for the 7 m antenna and (1.08,1.67) for the 30 m antenna. This indicates that especially for X2 the statistical variation is a result of the same process for both antenna heights. The superposition is also apparent in other data types although the bimodality of the statistics is not as clear. For instance, the number of unique ships observed per hour combines the effects traffic density and propagation variations making the separation of standard and anomalous propagation more complicated. From the number of ships, exp(Y), the anomalous propagation component Y2 of ship number logarithm for 30 m antenna was separated using quantile-quantile (QQ) plotting for the pair X2 and Y2. The QQ plot was linear for the quantiles corresponding to the half-distribution X2 from mode to infinity. This indicates that the values of parameter s are close to each other and are assumed equal, 1.68. The remaining two parameters of Y2 are obtained from a linear fit to the quantile plot. The results can be seen in Fig. . For the separated standard propagation part Y1 the Subbotin fit was less perfect, indicating that the statistics is dominated by traffic density variations. For the 7 m antenna the superposition was barely discernible in the ship number statistics.

To limit the over-forecasting of anomalous propagation, the statistical definition of the horizon in terms of the distribution model for X was chosen to be the 98th percentile of X1. For the hourly 95th percentile distance this is 82 km for the 7 m antenna and 94 km for the 30 m antenna (dashed line in Fig. ). This roughly corresponds to the estimations with a 60 m high transmitting antenna in Fig. . It is worth noting that for the 7 m antenna, the 98 percent cumulation of X1 results in a greater overlap of X1 and X2 (Fig. ) which means that the chosen horizon will likely under-forecast anomalous propagation when compared to the 30 m antenna.

For the hourly 95th percentile maximum distance over the 1-year study period, the frequency of OH observations was 34 % and 59 % for the 7 and 30 m receiver, respectively, while the corresponding numbers for the data south of Utö were 33 % and 62 %. From the superposition weights of X2, the selected horizon detects south of Utö 67 % and 91 % of the anomalous propagation instances for the 7 and 30 m receivers, respectively. For the 30 m antenna hourly ship number exp(Y) the values exceeding 37, or the value corresponding to the mode of Y2, are a clear indication of anomalous propagation in similar annual traffic conditions, detecting 50 % of the instances. This can be used as a simple indicator of anomalous conditions and occurs 23 % of the time.

The seasonal hourly occurrence of OH observations based on the hourly 95th percentile distance is shown in Fig. . The OH observations occur more frequently in spring and summer, where the hourly occurrence is up to 65 % for the 7 m antenna and up to 90 % for the 30 m antenna. During autumn and winter, the occurrence is lower, around 0 %–25 % for the 7 m antenna and 25 %–50 % for the 30 m antenna. A diurnal cycle where the OH observations occur more frequently in the evening and at night, and less frequently during the day, is also visible during summer. When the data is separated to the archipelago (north of Utö) and open sea (south of Utö), it is apparent that the diurnal occurrence of ducting is from the archipelago. In the archipelago sector, ducting occurrence increases 35 % from daytime to evening and night. Similar observations were made for the X-band coastal radar in Utö where the strong diurnal cycle was found to result from the archipelago sector where the marine boundary layer is influenced by the boundary layer over land .

The median RSSI over the study period is -110 dBi for the 7 m antenna and -109 dBi for the 30 m antenna (Table ). The hourly median RSSI varies between -120 and -60 dBi within-horizon, while over-the-horizon, it varies from -120 to -100 dBi. This is explained by the increased reception of weaker messages over-the-horizon. As such, RSSI alone is not a good indicator for AIS OH observations.

To examine the variability in received signal strength with respect to the distance from the antenna under standard and anomalous conditions, the RSSI over two months, September and October 2023, were plotted against distance, with the number of points illustrated in a 2-dimensional count histogram (Fig. ). October was chosen because it was identified as the month that had the least OH observations based on the 95th percentile time series (Fig. ) and a relatively stable number of daily messages (Fig. ), while September was identified to have frequent increased OH observations and variable number of daily messages. Only Class A and Class B position reports were included.

Although October was identified to have the least anomalous signal propagation, it is clear that there are still periods of anomalous signal propagation, although not particularly strong (c and d panels in Fig. ). To further assess this, received power (PRec) curve (Eq. ) was fitted to the October data with time periods of anomalous 95th percentile excluded and the data limited to the open sea (see Appendix for more details). The fitted PRec curve shows the received signal strength indicator (dBi) with distance under standard conditions.

The PRec curve fits well until the horizon (82 km for 7 m antenna and 94 km for 30 m antenna). When signal is received over-the-horizon, there is no straight-forward relationship between distance and signal strength. This is pronounced when comparing to September (a and b panels in Fig. ) when increased AIS range was frequently observed. It appears that over the horizon, the signal can travel to further distances without degradation, even up to hundreds of kilometers.

The AIS data from Utö antennae was compared with the ORBCOMM global AIS data to establish normal spatial coverage for the antennae. All data were limited to messages received from within the study area (see Fig. ) and to vessels with MID starting with numbers 2–7 (regional identifier for individual ships, e.g. first digit 2 stands for Europe). MID was used as the ORBCOMM dataset was very large and provided as separate files for each MMSI.

The AIS data from the two antennae was gridded into 0.25°×0.25° grid and the hourly visibility of each grid for September and October 2023 was calculated (Fig. ). Visibility for a grid is 100 % if at least one AIS message is received from within the grid every hour of the month. The extent of visibility increases with height and both antennae achieve great visibility along the busy ship routes. The visibility in September is much greater than in October. To address if the regions of low visibility are due to there simply being no vessels to receive messages from, or if the regions are out of range for the AIS antennae, the ORBCOMM global AIS data was also gridded into the 0.25°×0.25° grid and the visibility of each grid was presented for the two months (middle column in Fig. ). The global AIS data was then used as the “background” to estimate the Utö antenna's coverage area, and the antenna visibility was divided by the global AIS visibility for each grid. The ratios are shown as percentages (right columns in Fig. ).

Using the ORBCOMM global AIS as background is beneficial as it shows areas that can be expected to have constant coverage with the Utö AIS antennae (100 % visibility) and regions where coverage is occasionally achieved due to anomalous propagation (<100% visibility). However, near Utö visibility in some grids exceeded 100 %. The global data has a varying sampling frequency, depending whether the data is collected from the AIS base stations or from satellites. In addition, the probability of a successful reception of a single sentence message is higher than that of a multi-part message. If P is the probability, then for multi-part message Pn applies, where P is the probability of receiving a single message and n is the number of sentences in a multi-part message. For this reason, multi-part message reception probability is lower, especially in satellite reception of AIS data. As a result, vessels that appear in a grid cell for a short amount of time can occasionally be missing in the global dataset for that grid. This means that the visibility is likely biased higher and that the bias would likely increase if the grid size was decreased.

As October had the least amount of ducting, the grids with visibility >95% for October was then used to mask out the grids within the horizons of the AIS antennae. Each hour of the data was gridded and the number of grids with at least one ship during that hour was calculated over the study area for each hour (panel a in Fig.  and Appendix Fig. ).

The time series of number of grids over the horizon and 95th percentile of distance were compared. The correlation was high for both antenna heights, r=0.8 for 30 m antenna and r=0.74 for 7 m antenna (panel c in Fig.  and Appendix Fig. ). The 95th percentile of distances is more sensitive to the number of ships in the study area and their respective locations, e.g. a small number of ships during an hour could cause a spike in the hourly 95th percentile of distance, indicating a duct in a specific area.

In this study we have shown AIS OH observations. However, it has not yet been demonstrated in this study, that the OH observations result from ducting. Hence, the 95th percentile distance for the two antennae were compared to observed vertical modified refractivity gradients (M-gradients) calculated from the temperature and humidity profiles at Utö and to duct strengths calculated from the M-gradients (Fig. ). It is important to note that the maximum measurement height of 59 m a.m.s.l. limits the detection of elevated ducts. In addition, the mast is representative of local conditions, while ducts that could influence the signal propagation can exist within the horizon of the antennae, undetected by the mast measurements.

The time series show that when the 95th percentile distance was at its greatest during summer and autumn, a strong duct was also observed in the vertical M-profiles (Fig. ). However, it appears that the 95th percentile distance indicated OH observations more often, particularly over winter and spring, than a duct was observed in the vertical M-profiles.

To examine this more closely, the 95th percentile of distance was binned into 10 km interval bins and plotted against how often a duct was observed for each bin (Fig. ). It appears when 95th percentile is at the horizon (70–80 km for the 7 m antenna and 90–100 km for the 30 m antenna), a shallow duct (7–12 m) occurs 20 % of the time. When looking at the bins beyond the standard, the share of shallow ducts is smaller and the share of higher ducts (32–59 m) increases, and overall a duct occurs 20 %–40 % of the time. In addition, when the 95th percentile of distance is very low for the 7 m antenna, around 1–10 km, a shallow duct occurs 50 % of the time. Note that the 95th percentile can sometimes fall significantly below the defined statistical horizon for the 7 m antenna, which could result from e.g. the signal being physically blocked by the terrain.

Particularly, the highest 95th percentile distances seem to co-occur with the stronger and higher observed ducts. To examine this more closely, the occurrence of AIS OH observations were studied against the duct height and duct strength (Fig. ). When the duct height is observed at 59 m, the 95th percentile of distance is increased 90 % of the time for the 7 m antenna and 95 % of the time for the 30 m antenna while the share of stronger ducts also increases with the height of the duct. It appears that when the observed duct height is 32 or 59 m, AIS OH observations can be expected. Furthermore, the 7 m antenna observes less ducts than the 30 m antenna at all heights. However, the greatest difference occurs when the duct height is small. There are more obstacles to the 7 m antenna which can cause there to be less OH observations for the 7 m antenna when the duct height is low.

Although the occurrence of a duct with the height of 32 or 59 m seems to be a good indicator for AIS OH observations, the OH observations still often occur without an observed duct in the vertical M-profile. In other words, the measured M-profile under-predicts AIS OH observations. This is likely because the highest measurement height of 59 m might not be high enough for this purpose, and ducts affecting the AIS signal could have occurred above the highest measurement height. Furthermore, ducts within the AIS horizon, not captured by the measurement mast, could influence the AIS signal propagation.