Automating Weather Widget Generation

This entry is a fun little project I have been working on for the last week. It is nothing special or groundbreaking, but I hope you have some fun reading it.

One of the features we wanted to add to MadBoulder’s website (short description of the project at the end) was, for each page refering to a climbing area, a weather widget that provided info about the weather conditions and forecast in the area. This information is always useful when planning an outdoor activity.

The two most obvious approaches to add the feature are either querying a weather API and developing a nice layout to display the provided info or including a ready-to-use widget in the page. Since the second one seems more straightformward and less work,  it is the way I decided to go.

After taking a look at different options, I decided to use the widget provided by WeatherWidget.io . No registration nor providing an email required, I like how it looks,  – not a UI guy, but I think – it matches the website’s feeling and it is easy to setup.

Figure 1: WeatherWidget.io widget sample

The Problem

After having decided which widget to use, it was time to generate the widgets for all the supported areas and add them to the corresponding page. The most common way of doing so is navigating to the page from where the widget will be obtained, introducing the desired location, generating the widget code and copying and pasting it in the desired location in your webpage.

Figure 2: WeatherWidget.io widget generation page.

This is quite simple and not that much of a trouble if you only need a few widgets, but after repating the process 3 times it gets quite tedious and boring. We currently have 60 supported areas and the list is still growing. I did not want to have to generate each widget by hand. Ideally, I would like to be able to automatically generate the widgets from the list of current areas, which is something we already have.

The Solution

In order to avoid having to generate each widget manually I decided to develop a script that was able to:

1. Get the list of all the supported climbing areas.
2. For each area:
   1. Generate the code of the corresponding weather widget.
   2. Place it in the info page of that area.

In order to do so, we must inspect the code for the widget and be able to generate it ourselves.

Widget structure

Below I have copied the widget code I got after having set the desired location to Barcelona and pressing the GET CODE button:

<a href="https://forecast7.com/en/41d392d17/barcelona/" data-label_1="BARCELONA" data-label_2="WEATHER" data-theme="pure" >BARCELONA WEATHER</a>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src='https://weatherwidget.io/js/widget.min.js';fjs.parentNode.insertBefore(js,fjs);}}(document,'script','weatherwidget-io-js');
</script>

It can be seen that the widget has two parts, a link (a tag) and a script (script tag).

After a quick examination of the widget code, it is clear that most of it will be the same for any location. This parts of the widget code can be replicated without any modification.

The whole script part can be replicated as-is since it contains no information about the location we want to show the forecast of. Apart from that, most of the attributes in the link are also straightforward to generate. We only have to replace the location by the current location the widget refers to. With this in mind our “generic” widget template so far looks like this:

<a href="LOCATION_FORECAST_URL" data-label_1="LOCATION_NAME" data-label_2="WEATHER" data-theme="pure" >LOCATION_NAME WEATHER</a>
<script>
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src='https://weatherwidget.io/js/widget.min.js';fjs.parentNode.insertBefore(js,fjs);}}(document,'script','weatherwidget-io-js');
</script>

We just have to replace the tags LOCATION_NAME and LOCATION_FORECAST_URL with the appropiate values and voilà, the widget for that location will be ready to use. We can also change the theme by replacing the value of the data-theme attribute with the desired theme name.

Replacing the LOCATION_NAME by the location we want is trivial, but the same cannot be said for the LOCATION_FORECAST_URL . We need to understand how the url is built for any location to be able to generate it ourselves. If this is not done right we will get a 404 page as the one shown below:

Figure 3: 404 error page.

The good thing is that we can get as many sample urls as we want by generating widgets for different locations. And with enough data we should be able to reverse engineer the url build process.

URL Reverse engineering

The first thing to do is examine the different parts of the url and see what can we get from a quick glance. The url for the widget that shows Barcelona’s weather is:

https://forecast7.com/en/41d392d17/barcelona/

We can see that the url has two main parts. First we have the domain and what seems to be the widget language:

https://forecast7.com/en/

We can expect this part to be the same for any location. After that we have what looks like some location specific information:

41d392d17/barcelona/

Where the last part is trivial, since it just contains the name of the location. We can confirm our current hypotheses by checking the url obtained for another location (Valencia, Spain) and in a different language:

https://forecast7.com/en/41d392d17/barcelona/
https://forecast7.com/es/39d47n0d38/valencia/

Sure enough, it came out as expected. Now we just have to figure out what the alphanumeric sequences 41d392d17 and 39d47n0d38 mean. My guess was that this part of the url was being used to differentiate between locations that shared the same name. So, however it was generated, I was quite sure it had to be unique for each location even if they shared names. Testing a few more locations I ended up with the following list:

41d392d17/barcelona/
39d47n0d38/valencia/
46d428d84/chironico/
41d502d39/vilassar-de-mar/
46d127d02/salvan/

Furthermore, if we check the urls for different locations that have the same name:

41d392d17/barcelona/
10d14n64d68/barcelona/
26d22n103d43/barcelona/

It seems we have some sort of pattern. The only letters that seem to be present are d and n , and the number of digits doesn’t seem to exceed 3. A part from that, locations that are close to each other, such as Barcelona and Vilassar de Mar, have very similar sequences. This led me to think that what is being encoded in each of these sequences are the coordinates of the location. If we check Barcelona’s coordinates we can verify this guess:

Figure 4: Latitude and Longitude of Barcelona.

Bingo! As it turns out, coordinates are rounded to two decimals, d is used to replace decimal dots and n is used to replace negative signs. We already have all the required information to generate a valid url. The generic format is presented below:

https://_DOMAIN_/_LANGUAGE_/_LOCATION_COORDINATES_/_LOCATION_NAME_/_UNITS_

We already know everything we need to know to generate the widget code of any location. It is time to code the widget generator.

Automatic Widget Generation

Now that we have all the information required to build the widget it is time to code it. The structure of our program will be:

1. Get the location’s name
2. Get its coordinates via forward geocoding
3. Build the url
4. Test url validity and, if it is not valid, try to fix it
5. Replace the location name and url in the generic widget template
6. Place the widget in the page

The location is obtained from the area’s dataset provided by MadBoulder. To obtain the coordinates, ideally we would use the same service WeatherWidget.io is using, because if not there can be small discrepancies that will result an invalid url. However, I was not able to find from where they obtain the coordinates.

After testing different services, Open Cage’s forward geocoding API was the one that delivered the closest results and it is free to use. An excerpt of a sample response is shown below.

{
    [...]
    "results" : [
        {
            [...]
            "geometry" : {
                "lat" : 51.9526599,
                "lng" : 7.632473
            }
        }
    ],
    "status" : {
        "code" : 200,
        "message" : "OK"
    },
    [...]
    "total_results" : 1
}

We can then define a simple function that gets the coordinates of a given location:

def get_coordinates(location):
    """
    Given a location, retrieve its latitude and longitude
    coordinates via opencagedata API
    """
    location = location.replace(" ", "+")
    api_key = None
    with open("credentials.txt", "r", encoding='utf-8') as f:
        api_key = f.read()
    query_url = "https://api.opencagedata.com/geocode/v1/json?q={}&key={}".format(
        location, api_key)
    inp = urllib.request.urlopen(query_url)
    coords = json.load(inp)['results'][0]['geometry']
    return coords

Once the coordinates are obtained, we round the latitude and longitude to two decimals and build the expected sequence by chaining the latitude and longitude and replacing the dots and negative signs:

def format_coordinates(coordinates):
    """
    Given a set of coordinates in the form {'lat': LAT, 'lng': LNG}
    format them to weatherwidget.io expected url coordinate format.
    The format is:
    - coordinates rounded to 2nd decimal
    - dots replaced by d
    - minus signs replaced by n
    - LAT and LNG concatenated
    """
    coordinates['lat'] = round(coordinates['lat'], 2)
    coordinates['lng'] = round(coordinates['lng'], 2)
    lat = str(coordinates['lat'])
    if lat[::-1].find('.') == 1:
        lat += "0"
    lat = lat.replace(".", "d").replace("-", "n")
    lng = str(coordinates['lng'])
    if lng[::-1].find('.') == 1:
        lng += "0"
    lng = lng.replace(".", "d").replace("-", "n")
    return lat + lng

After that we are ready to build the complete url and the link tag of the widget:

A_TAG = """

    
    href="https://forecast7.com/_LANG_/_COORDS_/_LOCATION_/" 
    data-label_1="_LOCATIONPRETTY_" data-label_2="WEATHER" 
    data-theme="pure"
>
    _LOCATIONPRETTY_ WEATHER

"""

def get_url_location_name(location):
    """
    Transform the location name used to search the coordinates into
    the location format used in weatherwidget.io widget url
    """
    return location.split(",")[0].lower().replace(" ", "-")


def get_widget_code(coords, pretty_location, lang):
    """
    Generate the a tag of the widget from the retrieved info
    """
    location = get_url_location_name(pretty_location)
    tag = A_TAG.replace("_COORDS_", coords)
    tag = tag.replace("_LOCATIONPRETTY_", pretty_location)
    tag = tag.replace("_LOCATION_", location)
    tag = tag.replace("_LANG_", lang)
    url = "https://forecast7.com/_LANG_/_COORDS_/_LOCATION_/"
    url = url.replace("_COORDS_", coords).replace(
        "_LOCATION_", location).replace("_LANG_", lang)
    return tag, url

Now, since there might be some differences in the coordinates obtained from Open Cage’s API and the ones used by WeatherWidget.io, we have to make sure that the url we obtained is valid. To do so we can define a simple function that makes a request to the url and returns True if the request was successful and False otherwise.

def is_url_ok(url):
    """
    Test if the weatherwidget.io generated url is valid
    """
    try:
        req = urllib.request.Request(
            url, headers={'User-Agent': "Magic Browser"})
        urllib.request.urlopen(req)
        return True
    except urllib.error.HTTPError as e:
        print(e)
        return False

When the request fails, most of the time it will be due to some small differences in the coordinates. So, when the url fails we can try to fix it with an heuristic process that tests all possible coordinates in a given range centered around the given coordinates. We can increase or decrease the latitude or longitude by 0.01 each iteration and check if that change fixes the url:

def fix_url(coords, pretty_name, lang):
    if TOLERANCE:
        for i in range(-TOLERANCE, TOLERANCE + 1):
            nc = coords['lat'] + i/100
            for j in range(-TOLERANCE, TOLERANCE + 1):
                ncl = coords['lng'] + j/100
                formated_coords = format_coordinates(
                    {'lat': nc, 'lng': ncl})
                tag_code, url = get_widget_code(
                    formated_coords,
                    pretty_name,
                    lang)
                if is_url_ok(url):
                    return tag_code, url
    return "", ""

I found that a tolerance value of 5 will be enough to find the correct url most of the times. Now that we have all the different steps covered, our main function might look like:

def main(pretty_name):
    coords = get_coordinates(pretty_name)
    formated_coords = format_coordinates(coords)
    tag_code, url = get_widget_code(
        formated_coords,
        pretty_name,
        "es")
    if not is_url_ok(url):
        tag_code, url = fix_url(coords, pretty_name, "es")
    with open("template.html", "a") as f:
         f.write(tag_code + SCRIPT)

Where template.html in this case is just an empty html file used for testing purposes. After a few executions it looks like this which, when opened in a browser results in:

Figure 5: Widget generation results.

Final Notes

Sample Code

As always, you can find the sample code developed for this little project at GitHub .

MadBoulder

MadBoulder is a collaborative project that aims to become the reference library of boulder betas . Our idea is to collect as many betas as we can, from the easiest to the hardest.

The website makes it easier to navigate through the available information and find the desired video.

Thanks for reading!