The MIDAS data is downloaded from the BADC site as two files (hourly
Rain and hourly Weather, about 1.7 GB in all) for each year, containing
the data for all stations. This is then processed in the following
- The data for the required station is extracted, giving much smaller
files for each station and year of interest.
- The files are cleaned (a few hourly records are missing from some
files, and a few are duplicates). At the end of this step, every
hour is present and has exactly one line in the rainfall and weather
- The two files for each station and year are then merged, together
with data for sunrise and sunset times (generated on the fly as needed),
to form a simplified hourly file. This is a single file with
all pertinent meteorological data and daylight times, one line per
hour. At this stage there is nothing specific to flying in the data.
- The simplified hourly file is then processed against the flying
visibility and weather criteria, etc., to generate a flying bits
file – here, for each hour, single bits (0 or 1) are set to indicate
whether a given criterion is met (e.g., whether visibility minima
are met, whether the maximum wind speed in the hour is < 30 mph,
- Finally, the flying bits files are processed to generate the
individual analysis web pages for this website. This is done by
generating Memowiki » pages (pages
with Wiki markup) which are then converted to HTML before publication
to the website. Details about the station (name, county, etc.) are
extracted from the Met Office stations list file to help form the
All the generated pages are static in order to ensure good response
(there is a separate page for each combination of station, timeslot,
and year or range of years).
All of the above processes are effected by Rexx » programs and
are automatic (although some human intervention and inspection is
necessary to assure there are not too many missing records, etc.).
Please send any comments, questions, or corrections to me,
Mike Cowlishaw, at
All text Copyright © Mike Cowlishaw 2008, 2011. All rights
reserved. The pages and data here are for non-commercial use only.
This page was last edited on 2011-09-09.