The MIDAS data is downloaded from the BADC site as two files (hourly Rain and hourly Weather, about 1.7 GB in all) for each year, containing the data for all stations. This is then processed in the following steps:

  1. The data for the required station is extracted, giving much smaller files for each station and year of interest.

  2. The files are cleaned (a few hourly records are missing from some files, and a few are duplicates). At the end of this step, every hour is present and has exactly one line in the rainfall and weather files.

  3. The two files for each station and year are then merged, together with data for sunrise and sunset times (generated on the fly as needed), to form a simplified hourly file. This is a single file with all pertinent meteorological data and daylight times, one line per hour. At this stage there is nothing specific to flying in the data.

  4. The simplified hourly file is then processed against the flying visibility and weather criteria, etc., to generate a flying bits file – here, for each hour, single bits (0 or 1) are set to indicate whether a given criterion is met (e.g., whether visibility minima are met, whether the maximum wind speed in the hour is < 30 mph, etc.).

  5. Finally, the flying bits files are processed to generate the individual analysis web pages for this website. This is done by generating Memowiki » pages (pages with Wiki markup) which are then converted to HTML before publication to the website. Details about the station (name, county, etc.) are extracted from the Met Office stations list file to help form the pages.

    All the generated pages are static in order to ensure good response (there is a separate page for each combination of station, timeslot, and year or range of years).

All of the above processes are effected by Rexx » programs and are automatic (although some human intervention and inspection is necessary to assure there are not too many missing records, etc.).

Please send any comments, questions, or corrections to me, Mike Cowlishaw, at mfc@speleotrove.com.

All text Copyright © Mike Cowlishaw 2008, 2011. All rights reserved. The pages and data here are for non-commercial use only.

This page was last edited on 2011-09-09.