Google API-powered heatmap viewer of student visitor numbers at Newcastle University Open and Visit Days

As part of the School of Civil Engineering and Geosciences involvement in University student recruitment activities, prospective 6th form and college students can attend Open and Visit Days. These days give students the opportunity to come and learn a little bit more about the courses that are offered at the University, including those taught within the School. Within Geomatics, prospective students are given some experiences of what it might be like to study Geographic Information Science (GIS), or Surveying and Mapping Science (SMS) Undergraduate courses via a handful of taster exercises. These exercises are designed to enable staff members to talk about some of the basic concepts that a prospective student might learn about should they decide to apply and study GIS or SMS.

A key student recruitment activity within the School and more widely the University, involves the coordinated marketing and distribution of promotional materials focussed on Undergraduate courses to different colleges and schools around the UK.  In order to better understand how the School’s involvement in this activity leads to prospective students attending the University Open and Visit Days, thus showing an interest in the courses on offer from the School, a very simple web-based tool has been developed to record where prospective students are travelling from on Visit and Open Days, by recording against the school or college at which the student attends. However not only does this begin to allow recruitment staff to understand how marketing activities are leading to prospective students attending the Visit and Open Days, it also doubles as a taster exercise in explaining some of the basic concepts of data capture, management and visualisation that a student would learn more about within the GIS and SMS courses. A prospective student is able to search for the school or college that they attend from a geocoded set of more than 60,000 schools, and then subsequently increment a count against that particular school for the particular year in which they attended a Visit or Open Day. All this information is stored within a PostGIS-enabled PostgreSQL relational database, and is served out to the webpage via JSON following the use of standard SQL queries to query the underlying data. As a result a prospective student, as well as recruitment staff, are able to create custom Heat Maps (intensity, not temperature!), all powered by the Google Maps API, of their data, or data from previous years. The query interface allows different HeatMaps to be created based on sub-selections of school type, gender (boys only, girls only, or mixed gender schools) and years of interest.

For clarification the database stores no other information about the student other than a count against a particular school or college at which the prospective student attends, and the addition of new information is protected behind a username and password. The following images give some illustrations of this interface and tool:

Increment count against a school, at which a prospective student attends
Increment count against a school, at which a prospective student attends

 

HeatMap viewer, with criteria dialog
HeatMap viewer, with criteria dialog

 

HeatMap viewer, outputs
HeatMap viewer, outputs

ITRC Assembly, June 10th-12th 2013, Chilworth Manor Hotel, Southampton

During the early summer of 2013, the UK Infrastructure Transitions Research Consortium (ITRC) underwent a mid-term review, approximately two and half years after the inception of the research programme, which coincided with the annual ITRC Assembly. The assembly and review gave all of those working within the consortium, and also invited guests and delegates, the opportunity to hear about the work accomplished during the initial half of the research programme. The 5 year research consortium is funded through an EPSRC Programme Grant, with the mid-term review offering the chance to discuss the future of the flexible funding available for the final two and half years of the programme.

The three day meeting was held at Chilworth Manor Hotel in Southampton, and was facilitated by a facilitation group, Dialogue Matters to help coordinate and focus a delegation of researchers, academics, stakeholders and partners. Monday offered the chance for the Expert Advisory Group (EAG) to review the documentation and work completed under the five different work streams. Whilst part of this review took place behind closed doors alongside the program’s principal and co-investigators, subsequently the EAG gave direct feedback to all of those attending the three day meeting. This session was then followed by an open floor discussion and questioning by researchers, PhD students and investigators from the program of the EAG panel. The utility of the EAG in particular was felt by the program during these discussions, and also via their continued guidance on the cycle 2 and 3 assessments due for release in January 2014 and in the autumn of 2015 respectively. Further to the feedback delivered, the post-lunch slot was dedicated to researchers and investigators funded through the program the chance to present some more specifics about the tasks undertaken during the first half of the program. This was particularly effective in getting everyone up to speed with what others within the consortium had been working on, and helped certainly to set the scene for discussions about future directions assigned to day two and three. Finally, a group of ITRC-affiliated PhD students presented some scoping research they had carried out to try to pull together a set of data on projects, research centres and institutes at the global scale who are also working on similar research as that conducted under the ITRC banner. Not only was the presentation interesting in the manner in which it was delivered, the data and information collected offered a great starting point for further development of the ever-growing research community, acting as a focal point for information about the community at large.

Day two began to offer the affiliated researchers and investigated across the many universities represented within the program, the opportunity to address some of the following questions:

  • Where have we got to?
  • What is happening in this field, in other projects and around the world?
  • What externalities may adjust the way this research is conducted, or will influence the likely impact the research has e.g. changes in policy, education, funding, society, environment, markets etc)

Whilst discussions of these questions began immediately in the morning session to broaden the horizon of future possible directions, a selection of “seed” ideas or possible projects that were a priori selected as being potential key research directions were also considered. The opportunities to think more broadly about possible research directions for the final two and half years of the project and also consideration of ideas already identified as of interest, gave everyone the chance to give their opinion on what could or could not feasibly be achieved given the available remaining time and resource. From a personal perspective, I think this gave everyone a real sense of ownership of the future direction of the research and certainly helped to gauge the relative importance of the different tasks identified by researchers from wholly different backgrounds. Subsequently this session allowed researchers to consider new ideas and areas based on the knowledge gained during the first half of the program. The breadth of ideas was enormous, ranging from the need for autonomous analytics for infrastructure planning, provision, monitoring and recovery to the need for new systems to manage the proposed integration of unmanned aerial vehicles within commercially used airspace in the United States, currently being considered by FAA.

Whilst the majority of the second day was spent considering the future direction of program, the afternoon session gave an opportunity for those involved to take stock of the success of the mechanisms employed for internal communication within the consortium. As the consortium is spread over many research centres and universities, effective communication between them and within the consortium is critical to ensuring objectives are achieved. The qualitative review considered the utility of using social media to facilitate communication both internally and externally, such as the use of Twitter and Skype for external dissemination and internal discussions, whilst also appraising the use of the ITRC intranet for collaborative working, and assessing the state of the external facing ITRC website.

With Tuesday giving plenty of opportunity to widen the research agenda and look at possible future research directions that the consortium could move in to, as well as assessing what tasks are to be achieved within the remaining two and half years of the project, Wednesday’s agenda focussed on narrowing this scope. A series of research themes had been identified from Tuesday’s discussions, and researchers were invited to select a theme upon which to discuss what the key areas of interest within that theme might be. However, not only were ideas generated, but challenges to achieving success in these areas were also highlighted, to give an impression of the relative difficulty of each theme. The results of many of the discussions held on day two and three have certainly helped the principal and co-investigators of the program to coordinate what tasks and objectives are to be achieved within the final years of the program.

Overall the assembly and mid-term Review offered everyone involved in the program to take stock of the achievements to date, whilst recognising the significant challenges that lay ahead when trying to deliver on a program which is trying to understand the complex nature of infrastructure, how it is operated, and it’s likely resilience to impending changes in demography, economy and climate.

The following table offers a summary of those people who were involved in the three day meeting:

Role Name Affiliation
ITRC Expert Advisory Group (EAG)

Chairman

Colin Harris Independent
ITRC Expert Advisory Group (EAG)

Members

Rosemary Albinson BP
  Theresa Brown Sandia National Laboratories
  Jeremy Cooper Laing O’Rourke
  Yacov Haimes University of Virginia
  Geoffrey Hewings University of Illinois
  David Penhallurick HM Treasury
  Margot Weijnen TU Delft
     
EPSRC Representatives Christopher White EPSRC
  Iain Larmour EPSRC
     
ITRC Principal Investigator Professor Jim Hall University of Oxford
     
ITRC Program Manager Miriam Mendes University of Oxford
     
ITRC Investigators Dr Nick Eyre University of Oxford
  Professor John Preston University of Southampton
  Professor Chris Kilsby Newcastle University
  Professor William Powrie University of Southampton
  Professor Cliff Jones Newcastle University
  Dr Stuart Barr Newcastle University
  Dr Stephen Hallet Cranfield University
  Professor Pete Tyler University of Cambridge
  Professor Mark Birkin University of Leeds
  Dr Jim Watson University of Sussex
     
ITRC Researchers Simon Abele University of Oxford
  David Alderson Newcastle University
  Pranab Baruah University of Oxford
  Simon Blainey University of Southampton
  Modassar Chaudry Cardiff University
  Adrian Hickford University of Southampton
  Scott Kelly University of Cambridge
  Alexander Otto University of Oxford
  Raghav Pant University of Oxford
  Meysam Qadrdan Cardiff University
  Chris Thoung Cambridge Econometrics
  Rachel Beaven Cambridge Econometrics
  Martino Tran University of Oxford
  Chengchao Zuo University of Leeds
     
ITRC-affiliated PhD students Edward Byers Newcastle University
  Robert Carlsson University of Oxford
  Razgar Ebrahimy Newcastle University
  Timothy Farewell? Cranfield University?
  Ed Oughton University of Cambridge
  Oliver Pritchard Cranfield University
  Scott Thacker University of Oxford
  Katherine Young University of Oxford

ITRC WS1 Visualisation Workshop: Visualisation of multi-dimension data, 22/05/2013, St Hugh’s College, University of Oxford

As the ITRC programme progresses and approaches the mid-term review stage, in June and July of 2013, the of the work stream 1 (WS1) infrastructure capacity and demand modelling teams are beginning to produce outputs from their next round of modelling. Furthermore, the parallel development of spatial infrastructure networks as part of work stream 2 (WS2), is beginning to raise some significant challenges in terms of appropriate and effective data dissemination, communication and interpretation. The underlying high-dimensionality nature of the data being produced as part of WS1 for example, coupled with the complexity of the networks generated as part of WS2 means the consortium as a whole needs to begin to think about appropriate mechanisms to visualise these data.  For example, some initial prototypes of possible visualisation tools are beginning to be developed, (see here), but rather than build and design tools from the perspective of one researcher, it was considered more appropriate to consult with, other similar projects who are visualising similar data, or will require the ability to visualise similar data in similar ways to that required of ITRC, and also a host of visualisation and design experts from around the UK to gain better perspectives.

An initial workshop, organised by ITRC members, Dr Alex Otto (ITRC WS1 investigator), Dr Greg McInerny (Senior Research Fellow, University of Oxford), Mr David Alderson (Researcher in GeoInformatics, Newcastle University), Dr Stuart Barr (Senior Lecturer in Geographic Information Science, Newcastle University) and Miriam Mendes (ITRC Programme Manager, University of Oxford), sought to bring together relevant researchers from the plethora of Adaptation and Resilience in a Changing Climate (ARCC) network projects and leading researchers and experts in the field of data visualisation and design. Prior to the workshop, a questionnaire was distributed to both the invited ARCC project representatives and the visualisation experts in an attempt to give the organising team a better centralised perspective of what the respective groups would want to hope to gain by attending the workshop. The responses were then studied to tease out any overlaps between visualisation challenges faced across the ARCC projects, to attempt to collate a set of discussion points upon which to focus discussions in the afternoon of the workshop. Prior to these more focussed discussion sessions, the workshop initially allowed the ARCC project representatives to briefly (in 5 minutes or less) explain the nature of the project in which they are working, but also describe and explain some of the visualisation challenges being faced within that project. The aim of this early session was to allow the visualisation experts time to understand the background of the projects themselves, and also the nature of some of the data being produced, such that the more focussed discussions taking place in the afternoon had a little context.

From the responses to the questionnaire, and also following the morning’s ARCC project overview session, a series of 5 discussion topics were devised, that attempted to encapsulate the common visualisation challenges across all the projects, and are listed below.

  • Visualising multiple dimensions and scenarios;
    • Chair: Martino Tran (ITRC – University of Oxford)
    • Rapporteur: Craig Mills (Visualisation – UN)
  • The spatial dynamics of infrastructure networks;
    • Chair: Scott Thacker (ITRC – University of Oxford)
    • Rapporteur: Martin Austwick (Visualisation – UCL)
  • Temporal visualisation of infrastructure behaviour and response;
    • Chair: Sean Wilkinson (RESNET – Newcastle University)
    • Rapporteur: Min Chen (Visualisation – University of Oxford)
  • Simplifying and communicating effectively complex model outputs;
    • Chair: Jason Dykes (Visualisation – City University, London)
    • Rapporteur: Scott Kelly (ITRC – Cambridge University)
  • Multi-disciplinary co-production for infrastructure visualisation.
    • Chair: Simon Blainey (ITRC – University of Southampton)
    • Rapporteur: Jane Lewis (Reading e-Science Centre, University of Reading)

A chair and rapporteur, selected from the list of workshop attendees was devised such that each topic had a representative from the ARCC network, and from the visualisation community. Each topic was then discussed by attendees for about 10 minutes, with the chairs and rapporteurs capturing the salient points discussed around that particular topic. After 10 minutes of discussion the attendees subsequently moved on to the next discussion topic and a different table. Overall as a format for delivering break out sessions, this quick-fire, round-robin approach seemed to work well, allowing all attendees to discuss all the common discussion topics about visualisation, whilst at the same time having the discussions steered and reported by representation from both sides. The approach also seemed to help stimulate discussions between project representatives and visualisation experts, which was one of the objectives or organising and delivering the workshop. However further work is currently being undertaken to transform some of the excellent discussions in to a positioning paper with respect to visualising high dimensionality data for infrastructure planning and provision purposes. It is hoped that representatives from the projects, particularly those organising the workshop and on the ITRC side will be looking to further engage and collaborate with the visualisation community.  Watch this space…

Links to presentations split by those relevant to different communities are listed below:

Full Attendee List

ARCC Project-affiliated attendees (* speaker on visualisation challenges)

ARCC Project Representative Affiliation
ITRC Alex Otto* University of Oxford
ITRC Stuart Barr* Newcastle University
ITRC David Alderson Newcastle University
ITRC Raghav Pant University of Oxford
ITRC Scott Thacker University of Oxford
ITRC Jim Hall* University of Oxford (Principal Investigator – ITRC)
ITRC Miriam Mendes University of Oxford (Programme Manager – ITRC)
ITRC Simon Abele University of Oxford
ITRC Alex Leathard Newcastle University
ITRC Meysam Qadrdan Cardiff University
ITRC Modassar Chaudry Cardiff University
ITRC Simon Blainey University of Southampton
ITRC Kate Young University of Oxford
Transport Utilities’ Conversion Points (TUCP) Liz Varga* Cranfield University
All-in-One Tomasz Janus De Montfort University, Leicester
Undermining Infrastructure Jonathan Busch* University of Leeds
Land of the MUSCos Christof Knoeri* University of Leeds
Step-change Miles Tight* University of Birmingham
RESNET Sean Wilkinson* Newcastle University

 

Visualisation / Design Experts (presentations and speakers listed below):

Greg McInerny University of Oxford, Microsoft Research
Min Chen University of Oxford
Craig Mills United Nation Environmental Program World Conservation Monitoring Centre
Jason Dykes City University, London
Jane Lewis Reading e-Science Centre, University of Reading

 

Other invited attendees:

Vicky Hayman UK Climate Impact Projections, University of Oxford
Chris Cooper IBM, London
David Miller IBM, London
Mathew Carlos University of Oxford
Zoe Austin University of York
Martin Austwick UCL
Craig Robson Newcastle University
Glenn Hart Ordnance Survey
Paula Engelbrecht Ordnance Survey
Andrew Munslow Met Office

 

UK Infrastructure Transitions Research Consortium (UK ITRC) University of Oxford Newcastle University

Modelling change and adaptation in infrastructure systems: state-of-the-art modelling and simulation approaches @ TU Delft, Netherlands, 14th May 2013

As part of the geospatial engineering team’s on-going involvement in the ITRC project, researcher David Alderson was accompanied by newly-recruited Computing Science PhD student Mr Razgar Ebrahimy to attend a workshop kindly organised by Margot Weijnen, at TU Delft entitled “Modelling change and adaptation in infrastructure systems: state-of-the-art modelling and simulation approaches”. The workshop aimed to bring together researchers and academics from the Next Generation Infrastructures (NGI) team at TU Delft, representatives from across work streams 1 and 2 from ITRC (), and also welcomed the visit of Research Director of the SMART Infrastructure Facility at the University of Wollongong, Australia, Professor Pascal Perez, with the aim of sharing experiences, projects and outputs of each research team to help further build and maintain the research links and community developing between these three groups.

The morning session of the workshop consisted of a series of short presentations, each delivered by a representative of one of the afore-mentioned groups, with a view to then discussing some of the topics and concepts raised during the post-presentation discussion session. Initially we heard from Professor Paulien Herder about how research in to the current and possible future states of infrastructure systems should be considered as a combination of understanding both the traditional physical, technical components thought of when infrastructure comes to mind e.g. power stations, roads, water treatment works, but also the social actors that operate, maintain, build and ultimately consume services offered by infrastructure. Many studies of infrastructure systems tend to focus on the technical aspects of the systems, which are clearly of paramount importance to being able to deliver the levels of service society has become to expect when interfacing with infrastructure, but perhaps do not consider enough the impacts that “people” have on the performance and evolution of infrastructure over time.

The audience subsequently heard from Pascal Perez about the great work being undertaken at the University of Wollongong, Australia as part of the SMART Infrastructure Research Facility. A key aspect of the outcomes from Pascal’s presentation, was again the need to think about how the social actors play a role in the infrastructure “complex” system. This was of particular focus when thinking about the economic benefits of infrastructure service provision, and the conundrum as to whether it is society that drives the economic growth/decline in the first instance leading to either increased/decreased demand for infrastructure services, or whether economic growth/decline leads to changes in societal make-up and thus acts as the stimulus or suppressant for infrastructure demand and supply. The “chicken-and-egg” discussion point of whether it is the economy driving society or vice versa was of particular interest to Professor Peter Tyler (ITRC), Ed Oughton (ITRC) and Robert Carlsson (ITRC) who are interested in understanding the interactions between infrastructure and the economy, whether at a national, or regional level.

From a more technical perspective, the audience learnt about the excellent work being undertaken at SMART with respect to their development of a regional, SMART Infrastructure Dashboard, helping to enable infrastructure decision makers gain access to a plethora of infrastructure-related information via a tablet/mobile-compatible interface. The technical components, but also the design process through which this dashboard has been developed is of particular interest to work stream 1, 2 and 4, as it is the intention that something equivalent be prototyped and developed to potentially enable UK-based policy makers and planners, as well as scientists and researchers, to ability to access outputs from the various capacity and demand modelling activities from work stream 1, as well as looking at the possible infrastructure failures as part of work stream 2.

A mixture of further presentations by Pieter Bots, Igor Mayer and Igor Nikolic focussed more on the ways in which infrastructure systems and complex systems can be modelled and evaluated. In particular Pieter, thankfully, raised the point of ensuring that modelling, and the subsequent dissemination of results of those models, be tailored to suit the problem or challenge to be solved, but also tailored to the audience to which the results are being presented. A “one-size-fits-all” approach to dissemination is not an appropriate solution as the types of questions to be asked of complex systems by different audiences, may require different tools, techniques and visualisations suited to each audience. This ideal fits well with the approaches being thought about within ITRC, where tools developed will need to be adjusted to suit the needs of those using them. For example a three tier approach could be conceived to determine the functionality of any interface to infrastructure modelling data, whereby the highest tier offers access to information to a wider audience but allows decreasing levels of functionality and therefore reduces the complexity of questions that can be asked, against the lowest tier offering potentially more analytical capabilities but to, for example, only researchers within the relevant fields.

However, Dr Mayer’s presentation and discussion focussed on the potential application of gaming, or correctly termed, “serious gaming” to help evaluate the interactions between the social and technical aspects of the complex infrastructure system. For further information on some of the evaluation work, and other projects undertaken by Dr Mayer and his team, can be found here. It was particularly interesting to here how this approach allows the individual stakeholder to actually be immersed in the model or environment itself, and how their interaction or reaction to particular events or shocks can be evaluated as examples of how user’s interact with complex infrastructure systems outside the test environment. Dr Mayer raised a point heard previously when considering the use of games to evaluate socio-technical systems and that is to ensure that the appropriate level of abstraction from reality is applied such that a user does not become too disconnected from reality and as such does not interact in a manner as close to mimicking interaction with the “real” system as possible. This abstraction is important at both the functional level in terms of what functions any model is representing, but also in terms of the physical representation of real world features in a computer environment, whether that is in terms of selecting appropriate temporal and spatial scales across which to model a system. One drawback however, highlighted as part of the discussion sessions during the course of the day, was that the use of serious gaming to evaluate interaction can really only be performed a handful of times due to the restrictions of having to utilise people heavily in the evaluation process, and so this can make it difficult, to repeat evaluation activities.

Dr Nikolic helped to conceptualise the problem of complex adaptive system modelling, giving a great overview of what is really happening when a modeller models something. Dr Nikolic stated that any model, or any system is effectively a three-step abstraction from reality, with the level of complexity found within each of the following steps, increasing from left to right:

Computer Model <- Modeller’s Conceptualisation <- Stakeholder Understanding <- Reality

This was an interesting point to raise, and highlighted the necessity to include as many relevant stakeholders in the modelling design process to help capture as much of reality from different perspectives as possible. However, the audience agreed that stakeholder interaction and engagement, especially when thinking about the multiple actors involved in modelling complex infrastructure systems, can be one of the most challenging aspects of the modelling process.

Further during the post-lunch session of the workshop we heard about some great research efforts being undertaken in TU Delft to help develop tools and methods that can help in the modelling of complex systems. For example, Dr Gerard Dijkema, delivered a fantastic presentation on behalf of PhD student Chris Davis and others on the Enipedia database developed at TU Delft. The database contains relevant information on power generation facilities worldwide, gleaned by marrying together different linked open data sources available across the web. Clearly something of this nature is not only interesting in terms of a pure inventory of information, but is also of interest as a repository of information for energy-sector modelling purposes. The Wikipedia-style nature of the database, allows online users to edit information, as well as review different visualisations, plots, charts, maps, graphs of energy-sector information. These types of tools are now being used within TU Delft to help in further research work, and underlines the importance of trying to use consistent data sources for these areas of modelling activity.

Overall the workshop was a fantastic opportunity to see some of the research being undertaken within the Next Generation Infrastructure group at TU Delft, and subsequently further enhance potential collaboration opportunities between that group, UK ITRC and SMART, Australia. Many thanks to Margot Weijnen and her team for the invitation! It is likely that a similar workshop will be organised and hosted within the UK, for some time in 2014 to help continue building the links between the NGI, ITRC and SMART infrastructure research teams.

 

UK Infrastructure Transitions Research Consortium (UK ITRC) Next Generation Infrastructure Logo SMART Infrastructure Facility Logo

Linking OpenLayers, D3, JSON and NetworkX to build a topological and geographic network viewer

As many of the networks that I am building as part of my involvement in the Infrastructure Transitions Research Consortium (ITRC – www.itrc.org.uk) are inherently spatial, I began thinking about how it might be useful to be able to visualise a network using the underlying geography but also as an alternative, the underlying topology. I began exploring various tools, and libraries and just started playing around with D3 (d3js.org).  D3 is a javascript library that offers a wealth of widgets and out-of-the-box visualisations for all sorts of purposes. The gallery for D3 can be found here. As part of these out-of-the-box visualisations it is possible to create force directed layouts for network visualisation. At this stage I began to think about I can get my network data, created by using some custom Python modules, nx_pg and nx_pgnet and subsequently stored within a custom database schema within PostGIS (see previous post here for more details), in a format that D3 can cope with. The easiest solution was to export a network to JSON format as the nx_pgnet Python modules allow a user to export a networkx network to JSON format (NOTE: the following tables labelled “ratp_integrated_rail_stations” and “ratp_integrated_rail_routes_split” were created as ESRI Shapefiles and then read in to PostGIS using the “PostGIS Shapefile and DBF Loader”).

Example Code:

import os

import osgeo.ogr as ogr

import sys

import networkx as nx

from libs.nx_pgnet import nx_pg

from libs.nx_pgnet import nx_pgnet

conn = ogr.Open(“PG: host=’localhost’ dbname=’ratp_networks’ user='<a_user>’ password='<a_password>'”)

 #name of rail station table for nodes and edges

int_rail_node_layer = ‘ratp_integrated_rail_stations’

int_rail_edge_layer = ‘ratp_integrated_rail_routes_split’

 #read data from tables and create networkx-compatible network (ratp_intergrated_rail_network)

ratp_integrated_rail_network = nx_pg.read_pg(conn, int_rail_edge_layer, nodetable=int_rail_node_layer, directed=False, geometry_precision=9)

 #return some information about the network

print nx.info(ratp_integrated_rail_network)

 #write the network to network schema in PostGIS database

nx_pgnet.write(conn).pgnet(ratp_integrated_rail_network, ‘RATP_RAIL’, 4326, overwrite=True, directed=False, multigraph=False)

 #export network to json

nx_pgnet.export_graph(conn).export_to_json(ratp_integrated_rail_network, ‘<folder_path>’, ‘ratp_network’)

 

Having exported the network to JSON format (original data sourced from http://data.ratp.fr/fr/les-donnees.html), this was then used as a basic example to begin to develop an interface using D3 to visualise the topological aspects, and OpenLayers to visualise the spatial aspects of the network. A simple starting point was to create a basic javascript file that contained lists of networks that can be selected within the interface and subsequently viewed. Not only did this include a link to the underlying file that contained the network data, but also references to styles that can be applied to the topological or geographic views of the networks. A series of separate styles using the underlying style regimes of D3 and OpenLayers were developed such that a style selectable in the topological view used exactly the same values for colours, stroke widths, fill colours as styles applicable in the geographic view.  These stylesheets, stored within separate javascript files are pulled in via an AJAX call using jQuery to the webpage, subsequently allowing a user to select them. Any numeric attributes or values attached at the node or edge level of each network could also subsequently be used as parameters to visualise the nodes or edges in the networks in either view e.g. change edge thickness, or node size, for example. Furthermore, any attribute at the node or edge level could be used for label values, and these various options are presented via a set of simple drop down menu controls on the right hand side of the screen. As you may expect, when a user is interested in the topological view, then only the topological style and label controls are displayed, and vice versa for the geographic view.

For spatial networks, the geographic aspects of the data are read from a “WKT” attribute attached to each node and edge, via a WKT format reader to create vector features within an OpenLayers Vector Layer. It is likely this will be extended such that networks can be loaded directly from those being served via WMS, such as through Geoserver, rather than loading many vector features on the client. However for the purposes of exploring this idea, all nodes and edges within the interface on the geographic view can be considered as vector features. The NodeID, Node_F_ID, and Node_T_ID values attached to each node, or edge respectively as a result of storing the data within the custom database schema, are used to define the network data within D3.

At this stage it is possible to view the topological or geographic aspects of the network within a single browser pane. Furthermore, if graph metrics have been calculated against a particular network and are attached at either the whole graph, or individual node or edge level, they too can be viewed within the interface via a series of tabs found towards the bottom. The following image represents an example of visualising the afore-mentioned Paris Rail network using the interface, where we can see that the controls mentioned, and how the same styles for the topological and geographic views are making it easier to understand where one node or edge resides within the two views. The next stage is to develop fully-linked views of a network such that selections made in one window are maintained within another. This type of tool can be particularly useful for finding disconnected edges via the topological view, and then finding out where that disconnected edge may exist in it’s true spatial location.

Example of the geographic view of the Paris Rail Network displayed using OpenLayers (data read in from JSON objects, with geometry as WKT)

 

Example of graph metrics (degree histogram) for Paris Rail Network (data stored at network level)

 

Example of the topological view of the Paris Rail Network displayed using force directed layouts from D3.js

Open Source, Open Standards 2013 Conference, 18/04/2013, America Square Conference Centre

A member of the Geospatial Engineering team at Newcastle, David Alderson, recently attended a GovNet series conference in London, entitled “Open Source, Open Standards”. This was held at the America Square Conference Centre, and more information about the conference can be found here.

The conference delegates were largely comprised of various government agencies including the Department for Transport, Office of National Statistics, representation from emergency services, Department for Work and Pensions, Department for Education, Department for Environment, Food and Rural Affairs (DEFRA) as well as representation from many local councils from around the UK. From within these various organisations the delegates were largely found to be based within some part of their specific IT operations.

Keeping in mind the public sector background of a significant number of the delegates in attendance at the conference, many of the exhibitors were offering open source solutions to various IT-related challenges including content management, telecommunications, secure mobile offsite collaborative working, data storage amongst others. Some of the major players in the open source world were also exhibiting their products, and post-purchase services including redhat, and MySQL (Oracle), whilst there were also stands from (amongst others):

The conference overall was a fantastic opportunity for public sector employees, to gain further insight in to how open source solutions can offer alternatives to proprietary software, that are often found within government department and agencies as a result of a legacy of long-term IT contracts and vendor lock-in. The general feeling amongst those presenting was that open source offers IT managers, and those involved in the procurement process of IT within the public sector, fantastic competition to the proprietary software providers meaning that the options available are greatly increased and improved. However Tariq Rashid, Open Source Policy Lead, HM Government and a speaker at the conference was keen to stress that open source is not being “favoured” over proprietary solutions, and that both operate on a level playing field. The take home message for delegates seemed to be more related to understanding what open source can offer by dispelling fears and myths about it’s use or misuse, whilst intimating that the choice of open source vs proprietary should be more related to the problem to be solved, and that a mixture could be the best solution. 

A number of keynote presentations were delivered during the conference, and further information can be found at the conference website, including hopefully the presentation slides themselves. Of particular interest to geospatial people was the presentation delivered in the afternoon by Executive Head of Technology at the Met Office, Graham Mallin. He introduced some excellent work that has been undertaken at the Met Office with collaborations from other national meteorological services from France, South Korea and Australia, nearly all put together using open source products including the ever-popular Python products, Numpy, Scipy, Matplotlib, as well as GeoNetwork and PostgreSQL. A Space Weather interface was also briefly demonstrated during Graham’s presentation highlighting how open source is completely capable of handling all types of data and IT-related challenges.

Further information on the OpenWIS project can be found at the Met Office website.

Some other interesting talks were given by Mark Taylor, CEO of Sirius Corporation which operates as the UK’s leading Open Source services provider. Mark gave some interesting examples of how aspects of IT infrastructure within different organisations or government departments with which Sirius has worked with, were swapped or migrated to open source alternatives. The general take home message there seemed to be that caution is sensible and making the right choice for your problem is key, and that finally taking bite-size steps to replacing components is more sensible than wholesale change.

Upon reflection many of the speakers and exhibitors did a great job of promoting the use and exploration of open source alternatives at all levels of spatial and non-spatial software stacks but ultimately that the process of technology selection, deployment and maintenance is not that different to when considering purchasing licensed-based proprietary software.

Some interesting links:

http://scitools.org.uk/iris/docs/latest/index.html – IRIS tool

http://www.slideshare.net/zaiziltd/graham-mallin-head-of-it-infrastructure-at-the-met-office-presentation-at-the-open-gov-summit-2012 – OpenWIS explained by Graham Mallin, Executive Head of Technology, Met Office

From Newcastle to Darlington and back again…a tale of infrastructure failure.

Two members of the Geospatial Engineering team (David Alderson and Craig Robson) were due to present their current infrastructure/network-related research at the recent ITRC Early Career Researcher’s conference, held at Cambridge University on November 27th 2012. As such both embarked on a journey, departing from Newcastle at 0556 on the morning of the 27th, that would end having only reached as far South as Darlington…approximately 6 hours after departing! The cause of being only able to travel a few miles in that time…a flood-related failure of the rail network leading to a loss of power to the train and line between Durham and Darlington.  A set of images taken on the day of the failure illustrate the researcher’s plight.

https://twitter.com/dannysavage/status/273380288233631744/photo/1 – the train being towed (or not) by a DIESEL-powered engine

https://twitter.com/dannysavage/status/273383744277184512/photo/1 – the train being stuck…

http://ow.ly/i/1b2Ji – the water rises

The presentations that were due to be delivered are now available, along with a recording (cringe) of each presentation given by David and Craig.

Transport Planning Society: Applying GIS and Open Data in Transport

On a snow-covered evening, transport planners, policy makers and those generally interested in transport-related data, made their way to the School of Civil Engineering and Geosciences’ Cassie Building, for a Transport Planning Society organised event; Applying GIS and Open Data in Transport. The promise of tea, coffee, and the odd mince pie drew those in from the cold…only to discover that caterers had fallen foul of the snow and icey conditions, leaving the audience with little other sustenance than having to digest the two presentations delivered during the evening.

Initially we heard from CESER researcher Alistair Ford on the work he and colleagues in the School have been doing, looking at climate change mitigation and adaptation within an urban context. This work was conducted as part of the Geospatial Engineering team’s involvement in the Tyndall Cities programme, and the ARCADIA projects. The delivered presentation can be found here and here (click both links to retrieve all slides, the first link is just slides 1-20, and the second 20-37).

The audience then heard from Graham Grant from the Tyne and Wear Integrated Transport Authority (TWITA). Graham discussed the possibility of some UTMC-compliant data streams being “opened” to developers, potentially allowing access to approximately 5 pre-approved data streams, with some real-time data included. There are approximately 20-25 datasets that could be made open to developers, and so Graham (contactable at Graham.Grant@newcastle.gov.uk) and colleagues will be looking for developers to prioritise that list, and help select data of most interest to them.

UPDATE: The list is now available: UTMC Data Sources – Provided by Graham Grant.

Thanks to Laura Hanson (CEG), and Nicola Hill (Senior Transport Planner, ARUP) for organising the event. Find out about more Transport Planning Society events here.

 

ITRC Stakeholder Workshop, The Royal Statistical Society, London, 6th November 2012

The second ITRC Stakeholder Workshop was held at The Royal Statistical Society, on November 6th 2012. The workshop presented an opportunity for project partners, stakeholders and the Expert Advisory Group of the ITRC project to hear about progress on the main work packages from a wide range of researchers and investigators, and again offer their opinions and feedback as to what aspects of the programme they would like to know more about, or become further engaged with.

Those attending the workshop, included representation from industry, government and academia who participate in infrastructure provision, maintenance, delivery and research, including:

Government / Pseudo-government Research / Academia Infrastructure / Utility Providers Engineering Consultancies Local Government Other
Committee on Climate Change University of Illinois at Urbana-Chamaigne BP International Atkins Hampshire County Council Institution of Civil Engineers
Department of Energy and Climate Change ECI, University of Oxford United Utilities Black & Veatch CIRIA
Department for Transport University of Cambridge Northumbrian Water Ltd JBA Trust
Highways Agency Imperial College London Thames Water John Dora Consulting Limited
Climate UK University of Southampton National Grid Bam Nuttall Ltd
Environment Agency University of Oxford COSTAIN Group
DEFRA University of Leeds Parsons Brickerhoff
HM Treasury University of Newcastle AECOM
EPSRC University of Exeter CH2M HILL
National Soil Resources Institute, Cranfield University MWH
Sandia National Laboratories
University of Birmingham
TU Delft, Netherlands
University College London
Judges School, Cambridge
University of Wollongong, Australia
University of Sussex
UKCIP, University of Oxford

 

During the workshop, presentations were delivered by various members of ITRC, including the following:

–          ITRC Infrastructure Strategies (WS1): Alex Lorenz (Oxford University)

–          Process for conducting the next cycle of ITRC assessment (WS5): Robert Nicholls (Southampton University)

–          Infrastructure Risk Analysis and the ITRC Infrastructure Database (WS2, WS4): Raghav Pant (Oxford University), Stuart Barr (Newcastle University)

One of key aims of the workshop was to help guide aspects of WS1, and consult stakeholders about reviewing and identifying infrastructure options and begin to build cross-sector strategies. This process was coordinated by Alex Lorenz (WS1 Researcher) and Roger Street (ITRC Stakeholder Champion), and required stakeholders to review the proposed infrastructure options from ITRC within each infrastructure sector (energy, water / waste, transport) for their completeness and priority for evaluation. The options proposed within each sector represent a change or alternative to the current status, possibly in terms of structural change, or influencing supply and/or demand. For example, consideration of the electrification of transport could be considered an option within the energy sector. Furthermore, this led to consideration of cross-sector strategies whereby options from different sectors can be considered together e.g. energy-transport options, or energy-water options for example.

The workshop also presented an opportunity to showcase the custom network database schema developed within Geospatial Engineering @ Newcastle, and the Python linking architecture coupled to that. This system is being used to build national and regional-scale infrastructure network models to then subsequently form part of the infrastructure risk analysis being carried out in WS2. If you would like to read more about this, please see a previous post here.

Further to the afore-mentioned, ITRC-specific presentations, the audience also heard from two external speakers, who delivered keynote addresses on complex adaptive system modelling, and future finance options for infrastructure provision:

–          Complex Adaptive Systems modelling to inform policy decisions: Theresa Brown (Sandia National Laboratory)

–          New business models for national infrastructure provision: Simon Taylor (University of Cambridge)

Stakeholder workshops such as this, provide project partners with a direct opportunity to get involved in the various aspects of the consortium, and help to continue the consortium’s success to date with stakeholder engagement and co-production, a key deliverable and consideration of WS5. This process will continue with the next cycle of assessment as part of WS5.

Geospatial Engineering @ 4th Annual Open Source GIS Conference, University of Nottingham, September 5th 2012

The 4th Annual Open Source GIS Conference, held at the University of Nottingham on September 5th this year was attended by four members of the Geospatial Engineering team at Newcastle; David Alderson, Tomas Holderness, Alistair Ford and Craig Robson. The conference offered an opportunity for those working in academia, government and the private sector to listen to some of the latest technological and research developments focussed around open source GIS.

During the conference, David Alderson, Research Associate in GeoInformatics delivered a presentation focussed around the network modelling framework being developed at Newcastle University, entitled “An open source relational database schema and system for the analysis of large scale spatially-interdependent infrastructure networks”. The framework links the network analysis Python package, networkx, with a PostGIS database, allowing a user to take raw point and line data, and convert them in to topological and spatial networks. Additionally the framework, and database schema, enable the interdependencies between different networks to be represented and stored. The modelling of interdependencies between different infrastructure networks e.g. between gas and electricity, or water and electricity, will help to understand how infrastructure networks are dependent on resources supplied from a different network. This work has been a collaborative undertaking between researchers, and PhD students within the Geospatial Engineering team at Newcastle.

The slides delivered at the presentation are here.

The conference also offered an opportunity for the team to attend a workshop on the open source software “GeoNetwork”, with a view to potentially utilising this as an alternate front-end to the Geomatics Data Server (GDS) developed at Newcastle University. The GDS was developed to act as a metadata server, and data repository for data held by colleagues working within the School of Civil Engineering and Geosciences. The system allows users to add and search metadata records, stored within a PostGIS database. Currently the system is used by researchers, 3rd year undergraduate project and PhD students, to help assist them in finding data for their research. The GeoNetwork workshop gave the GDS developers some basic knowledge of the capabilities of the software, and some more detailed tutorials and materials to help further explore whether the software will be suitable for the GDS.

The paper is available here.