GISRUK 2014 – Céilidhs, kilts and k-means cluster analysis

From the 16th April to 18th April I attended GISRUK in a surprisingly sunny Glasgow. The started with a welcome from Jane Drummond followed by an interesting key-note talk from Mike Worboys , A Theoretician’s eye view of GIS Research. He talked about how GISRUK has developed there has been  a dramatic fall in the proportion of papers that covered the theoretical side of GIS, with the focus now being on Application.

Talks from the first day that I particularly enjoyed focused on Spacebook, a system of delivering directions via audio as users encountered various way points on a route. William Mackaness talked about his research in comparing directions given using street names and direction given using landmarks.

Phil Bartie, who was a researcher on William Mackness’s paper delved deeper into the issue of Landmarks. Using images he looked at what people identified as landmarks and then analysed them semantically and spatially to distinguish related and unrelated features.

The following day saw me presenting a talk on some of the sensor infrastructure work we’ve done at Newcastle using HStore as a key-value based approach to storing heterogeneous data. Robin Lovelace’s talk round the merits of twitter data in research. Sparking some interesting debate about how research using twitter data uses data from the public so should benefit the public.

Thursday evening then featured the conference dinner followed by the Céilidh, an event that I was secretly dreading, dancer I am not. So was pleasantly surprised at how much fun the event was, as someone pointed out to me; it’s dancing but dancing with an algorithm.

Friday morning then featured my favorite talk of the conference with  Paul Brindley talking about his Phd work on extracting the additional address information from addresse listed on the internet to map out neighborhoods. A simple idea but with fascinating results.

The conference was then wrapped up with a keynote from Professor Alias Abdul Rahman who gave an interesting overview on the state of 3D GIS work.

The event was a fun filled thought provoking and thoroughly interesting. Thanks must  go to Jane Drummond for seamlessly organizing the event, even managing the weather. I would also like to thank William Mackness who brilliantly chaired my session.

Bring on Leeds 2015. Wonder what there version of Céilidh will be?

Neil – @neil_py_harris

EuroSDR EduServ E-Learning 2014

Monday 3rd till Tuesday 4th March saw the introductory lectures held for the 12th EuroSDR EduServ e-learning course at the University of Trento, Italy. Located just south of the Alps, the city sits in the Adige Valley so is surrounded by snow-caped mountains to give a picturesque view. The lectures were attended by a small delegation, mainly from national mapping agencies and universities from across mainland Europe. The four courses of the EduServ programme were introduced and presented by the various course leaders over the two days.

Trento University

View from Bruno Kessler Foundation, Trento University

On the first day Professor Norbert Haala, of Stuttgart University, started proceedings and presented material for his course on high density imaging matching. He presented some of the results of DSM creation from different software packages as part of his EuroSDR benchmark on image matching, whilst giving an overview of the Semi-Global Matching (SGM) algorithm, which I am using as part of my PhD, It was shown how a dense point cloud can be created from UAV and aerial photography by using the SGM approach. Hopefully this workshop will give greater insight and help overcome issues that have been experienced with ‘noisy’ photogrammetric point cloud.

Afterwards Dr Petri Ronnholm, of Aalto University in Finland, presented his course on the integrated use of airborne laser scanning and aerial photogrammetry. This was again based around a EuroSDR benchmark which tested different methods for the integration of the two dataset, some of which will be used in the course. The lecture was concluded with an interactive session to discuss the advantages and disadvantages of integrating the two dataset and what future applications this could be used for. One point that arose and was discussed was whether there was a need to integrate the two datasets due to high density point clouds being created from imagery, by the principles outline previously by Professor Haala. It was concluded that although lidar is still essential for forestry application, research may prove that photogrammetric point clouds are just as suitable as lidar for other applications.

A meal was held after the first day’s lectures at a local restaurant in Trento, with exquisite pasta and other traditional Italian food enjoyed over four courses and wash down with a glass (or two) of local wine. This offered a great ice breaker and a way to get to know other delegates as well as the course leaders.

Dr Clement Mallet, of IGN France, started the second day by presenting material for his course on change detection in high-resolution land-use/land-cover geodatabases and presented work from his EuroSDR benchmark regarding change detection methods. The need for land cover and land use was introduced followed by many different approaches presented, which mainly used satellite imagery.

Dr Daniella Poli, of Terra Messflug GmbH, closed proceedings with the last of the four courses on mapping using high-resolution satellite imagery. An insightful overview was given of low resolution film-based satellites to new high resolution digital-based sensors as well as the processing that is required for processing the data. This carried on from some of the principles covered by Dr Mallet, giving more details on the processing that may be required in order to use satellite imagery including radiometric corrections and Rational Polynomial Coefficients.

Thanks to Fabio Remondino of Trento University for hosting an excellent introductory workshop. With the first course starting today (10th March) and the final course finishing on the on the 13th June it is hope many new skills will be learnt over the next four months.

 

Andrew McClune

PhD Student in Photogrammetry

A summer of SALCA

 

Geospatial Engineering researchers recently took a trip ‘Down Under’ to participate in a unique terrestrial laser scanning inter-comparison exercise. Dr Rachel Gaulton and Dr Steve Hancock joined almost 30 leading researchers from the UK, Australia and the U.S. at field sites near Brisbane in early August to evaluate and compare how five different laser scanners and a range of other measurement approaches can help to measure and monitor forest canopy structure. These scanners included the Salford Advanced Laser Canopy Analyser (or SALCA) , the subject of on-going NERC-funded research at Newcastle, in collaboration with University of Salford and UCL, examining the potential of dual-wavelength laser scanning for assessing forest canopy health.

The exercise, organised by John Armston and colleagues at DSITIA Queensland and CSIRO, was an activity of the Terrestrial Laser Scanning International Interest Group (TLSIIG), a recently formed network of scientists with an interest in forest laser scanning. Alongside SALCA, measurements of three field plots were made with the World’s only other dual-wavelength TLS, DWEL (developed by Prof. Alan Strahler at Boston University with collaborators at CSIRO, UMass Boston and UMass Lowell), the low-cost Canopy Biomass Lidars (named Candi and Bruno and developed by UMass Boston and RIT) and two commercial systems. Photogrammetric techniques also played a part with co-incident measurements made with the AusPlots ‘Photopoint’ method, a system designed to obtain 3D forest structure information from photo panoramas. A report on the exercise featured on Australian TV News .

SALCA 1

Intercomparison exercise participants and their scanners. From left to right: A Riegl VZ400, a CBL, DWEL, a Faro scanner and SALCA.

Alongside the high-tech methods, leaf samples from the tree canopies were needed to allow measurement of spectral properties and to test the ability of dual-wavelength data to distinguish leaves and bark – a key factor in producing accurate estimates of canopy structure. Steve joined the destructive sampling team in using a ‘line thrower’ (or big slingshot) to collect the samples and undertook additional sampling to estimate the true leaf area index of a section of heavily laser scanned tree canopy.  Work is now on-going to calibrate and compare the data sets, with early results presented by TLSIIG members (John Armston, Crystal Schaaf and Alan Strahler) at the Silvilaser 2013 conference in Beijing.

Ian Paynter (UMass) and Steve with canopy samples and the line thrower (left) and a subset of SALCA data from Brisbane Forest Park (right).

The work in Brisbane followed a month-long field experiment with the SALCA instrument at the University’s Cockle Park Farm.  The experiment, part of a NERC-funded research project examining the potential of dual-wavelength laser scanning in forest health monitoring, was focussed on testing the sensitivity of the instrument to changes in canopy water content – an indicator of drought or disease. The canopy-scale experiment at Cockle Park involved subjecting 22 trees to drought stress, whilst making extensive physiological, spectral and laser scanner measurements and preliminary results have recently been presented by Rachel at the RSPSoc 2013 Annual Conference in Glasgow and the 9th EARSeL Forest Fire Special Interest Group Workshop held in Warwickshire.

More information about the SALCA instrument and on-going research can be found in the SALCA Diaries.

Scanning small-leaved lime at Cockle Park farm and SALCA data from a group of trees suffering drought stress (colours indicate reflectance at 1545 nm).

Many thanks are due to John Armston and colleagues for their hard work organising the Brisbane field work. Steve Hancock’s travel to the inter-comparison exercise was funded by a small grant from the Douglas Bomford Trust.

Football Tweets Part 2 – Local tweeter or Glory tweeter

A while ago here at Newcastle we set up a system to record the locations of football fans on twitter. Whilst this system was mainly a bit of fun and also a great means of testing our infrastructure it also very rapidly provided us with a considerable amount of data. And with an upcoming talk about our football tweet work I was given the opportunity to again carry out some analysis on this data.

Last time round when I analysed the football data I used about a month’s worth of data to look at football team fan locality. However this selection of data wasn’t particularly fair as it contained a number of different fixtures with teams playing both home and away which would have heavily affected the distance from tweet to club. For instance if Newcastle were playing away at Fulham it’s not fair to measure the distance from “#NUFC” tweets to St James Park “ooh look Newcastle have loads of fans in London, they’re not local supporter are they”. So this time round I looked at just tweets taking place in the UK during the recent international break during which there were no Premier league games. The map below shows this data subset.

Firstly I carried out the same analysis as last time whereby I measured the distance from tweet to the ground of the club it was about.  These distances were then averaged per club to give an average tweet distance for each club. The result are below (click to enlarge). The club with the shortest distance was West Brom with a very impressive average distance  of 5.7km. However we only actually recorded 10 tweets during this period, so in short not many people tweet about West Brom but the ones that do are very close to The Hawthorns. At the other end of the spectrum you have your expected “glory” clubs. Your Liverpools, your Man Us and your Norwich Citys…

…hang on Norwich City?? I myself am a Norwich City fan so found this stat at little hard to believe, you’d be hard pressed to call me a glory supporter. I tried to think about why Norwich may have scored so highly here. My conclusion was that as Norwich is the only football league team in Norfolk it represents a larger area than most clubs. Therefore this large distance could maybe be justified.

So my next piece of analysis was to look at whether the tweet about a club fell in the same county as that club. Again my results are shown below. Yet again West Brom performed the best with 100% of its tweets falling in the west midlands. And Norwich city had disappeared from the bottom 3 into mid table (something I wish we’d do in the premiership). But now the worst performer was Hull City.  Had their rebrand to Hull City Tigers really caused them to have a wider fan base? Probably not, this is probably caused by Kingston upon Hull being considerably smaller in comparison to a lot of other football team counties. And you could very easily be from outside Kingston upon Hull with Hull city still being your nearest (premier league) club.

Therefore I thought I’d carry out another piece of analysis this time looking at whether or not the tweet was about their nearest club.  Once again my results are displayed below.  Here Hull have leapt from bottom to 2nd and Southampton have also made a considerable leap up the table.  However again I noted something from the results which was that this time the bottom 5 is made up of clubs with another in very close proximity so the tweet may still be about a “nearby” club but not be counted as there is a club closer.

To account for this I needed some meaningful distance which would be considered as local. After a quick search I found that CAMRA consider any ale produced in a 20 mile radius of the pub to be local, could this be applied to football fan? Therefore I conducted one last test using this CAMRA metric of “localness” which counted the number of tweets which had no closer club OR were with 20 miles of the club.  And for a final time my results are shown below.

Hopefully this shows some interesting results produced by just a few simple POSTGIS queries.

Neil – @neil_py_harris

Strewth, these guys are SMART!

The inaugural International Symposium for Next Generation Infrastructure was held between 1st and 4th October 2013, at the SMART Infrastructure Research Facility at the University of Wollongong, Australia, and one lucky Geospatial Engineering researcher from Newcastle was able to attend. David Alderson gave a 20 minute presentation entitled A National-Scale Infrastructure Database and Modelling Environment for the UK following a successful submission of a conference paper to the conference committee, under the same title. The work contained within the paper and the presentation represented an amalgamation of work conducted by David and other researchers from the Geospatial Engineering group at Newcastle and other research institutes and universities involved in the Infrastructure Transitions Research Consortium (UK ITRC) programme. The focus of the paper and presentation was to give readers and delegates alike a glimpse of some of the work undertaken in the process of constructing a database of infrastructure-related data relevant to the UK. This included not only an overview of some of the datasets that may be found within the database, but also a preview of some of the visualisation tools that are being developed on top of the data. An overview of these visualisation tools can be found within other posts in this blog site here.

A copy of the slides can also be found here and here. Unfortunately the presentation has had to be split into two parts, so please download from both links to get the full presentation.

Other representatives from the School of Civil Engineering and Geosciences at Newcastle University, UK could also be found delivering presentations at the event including:

Miss Sarah Dunn, PhD Student – Modelling infrastructure systems for resilience and sustainability (part one, part two)

Mr Shaun Brown, PhD Student – Resilience of resource movements to disruptive events

Mr Matthew Holmes, STREAM PhD Student – How do we ensure the assessment of infrastructure resilience is proportionate to the risk?

Further to this fantastic opportunity, a further round of meetings looking to develop collaborations between researchers at SMART, including former Newcastle-based PhD student and post-doctoral researcher Dr Tomas Holderness, and the Geospatial Engineering group at Newcastle, is being held at the SMART infrastructure facility between October 8th and 11th 2013. These meetings will look to focus on potential collaborative opportunities regarding network interdependencies between infrastructure networks, and also web-based data dashboards for visualisation and dissemination purposes.

Watch this space for more information…

Infrastructure dashboard prototype: Economics and Demographics Data Dashboards

In an attempt to begin to think about how some of the data being generated by colleagues within the Infrastructure Transitions Research Consortium (ITRC) could be visualised, I started by considering how to visualise the seed projections that are used as inputs to the infrastructure capacity and demand models (CDAM) that have/are currently being developed by different institutes within the consortium. To find out more information about this aspect of the consortium please follow this link. The demographics projections have been developed by Leeds University, and give indications as to the number of people living in each government office region and local area district within the UK, out to the year 2100. Alongside this standard variable, projections for the quantity of urban area and number of households are also calculated and provided, although at this stage they are not being fed in to the CDAMs. The economics projections, developed by Cambridge Econometrics, contain many variables with reference to employment levels, imports/exports, gross value added (GVA) by industry, energy use by different fuel users, and many others. Each of these variables is supplied along a similar timescale to the demographics projections, whilst some are also disaggregated from UK-wide values to the government office region level. The following variables from the economics projections are supplied with this spatial disaggregation:

–          GVA by region and industry

–          Household expenditure by consumption category and region

–          Investment by investing sector and region

–          Employment by industry and region

The two data sets themselves are supplied in either comma-separated value (.csv) format, or via the use of the netCDF (.nc / .cdl) formats. In order to store these data, and then subsequently query them, the data is loaded via a Django-powered web form in to a PostGIS-enabled PostgreSQL relational database. The web form uploading process attempts to validate the values supplied within each of the projection sets against sensible ranges of values for each variable such that any erroneous data values can be detected. This approach ensures that the data is stored alongside other data related to the consortium, such as the infrastructure network models, and subsequently makes it quite (!) straightforward to create a web-based visualisation dashboard / platform for the data. The chosen web framework, Django, is Python-based and natively allows connections to PostgreSQL/PostGIS-enabled databases.

Both sets of projections, demographics and economics, are similar in nature in that they have a mix of UK-wide outputs over time, as well as a mix of spatially disaggregated outputs. I therefore chose to include the ability to view both a map, as well as charts and plots to give a user access to as much of the information as possible, without overloading them, within a single web page. In order to achieve this, a series of Django-enabled, and just standard SQL queries were developed to deliver data from either of the projections sets as JSON to the webpage. The webpages themselves (one each for demographics and economics respectively) communicate to Django via a synchronous AJAX request, which is all handled and powered via the use of the JavaScript framework, jQuery. Each webpage offers the user a series of dimensions or options that they can choose from, resulting in a new request being sent to the server to retrieve data matching against the chosen criteria. The following table represents the options available to a user for the two different interfaces:

Demographics Dashboard Economics Dashboard
Data:–          Demographic projections – number of people in each government office region, or local area district

–          Proportion of urban area projection – the percentage of each government office region, or local area district considered as urban

–          Change in the proportion of urban area projection – the change in the percentage of each government office region, or local area district considered as urban.

–          Household projections – the number of households in each local area district

 

Data:–          Economics projections – a set of different economic-focussed variables in relation to employment, consumption, energy use and much more
Scenario – a user can select from a drop down list of scenarios that have been uploaded to the database. Each set of demographic data represents a different scenario of demographic change, and each value within the database references which scenario to which it is related. The different scenarios are generated by using different values for the attractors and detractors that make a region or district more or less attractive to reside in. Scenario– a user can select from a drop down list of scenarios that have been uploaded to the database. Each set of economics data represents a different scenario of economic change, and each value is referenced against a particular scenario. In the case of the economics projections, each scenario represents a different combination of inputs for:-          Population-          World Economic Growth

–          Fossil Fuel Price

Time – a drop down list of available years of data Time – a drop down list of available years of data
Location – depending on the user’s selection of scenario, a drop down list of government office regions or local area districts is presented. The selection of a particular location is also possible via direct interaction with the map interface itself. Location – as previously described, only a subset of all economics variables are disaggregated to the government office region level, and these regions are presented via a drop down list. Similarly to the demographics data dashboard, a location of interest can be selected by interaction with the map interface itself, as well as via the drop down list.
Number of equal-interval map classes – this option allows a user to determine the number of equal-interval classes to use when mapping the data satisfying the matched criteria defined by the user.
Adjust overlay transparency – this slider allows a user to increase or decrease the transparency of the overlay map displaying the projections to allow them to more easily see the underlying base map layers provided. This for example can be used to help the user orientate themselves on a particular area of the UK, before seeing the overlay of chosen demographics or economics values.
Gender – when selecting the standard demographics output i.e. the number of people in each region or district, this option determines whether the displayed map is showing projections for males, females, or both i.e. the total population. Variable– the user is able to select from a drop down list of 18 possible variables to retrieve maps and charts about. The particular choice of variable determines which combination of the following options can subsequently be selected:-          sector

–          regional industry

–          fuel user

–          fuel type

–          consumption category

–          UK-wide investment sector

–          Regional investment sector

–          UK macro economic indicator

–          region

 

Age category – similarly to the gender option, when selecting the standard demographics output, this option determines which age category is being mapped. Each category represents a 5 year cross-section of the population, ranging from 0 to 90+.
Normalisation – this option allows a user to tell the displayed map to create equal-interval classification bounds based on other values that also satisfy the user’s selected criteria. For example a user could wish to normalise based on values for males/females for a particular region or against the whole of the UK.

 

As a user is changing the combination of the afore-mentioned options selected, whether for the economics or demographics data projections, a new request is sent to the server to query the database. Once the JSON data matching the selected criteria is returned to the webpage, JavaScript alongside the use of the HighCharts JavaScript-based graph library are employed to create the charts and graphs of the time series data. I selected HighCharts as it offers some fantastic, out-of-the-box functionality such as the slick animations activated when new data is supplied, or the hover-over tooltips to return actual data values. HighCharts however is a paid-for product, but has been used here as the work falls under the academic/research use. Furthermore, it is pretty straightforward to coerce the JSON output from the database in to something that HighCharts can handle. The maps themselves are created via the use of the OpenLayers JavaScript mapping client, largely employing the client-based vector layers and features for the rendering, rather than employing a WMS to serve the data. However it is possible that in the future a WMS version of the data streams will also be needed, probably employing Geoserver to do the leg work for this.

The following images illustrate the demographics and economics dashboards in their current form:

Demographics Data Dashboard

Demographics Data Dashboard

 

Economics Data Dashboard (household consumption)

Economics Data Dashboard (household consumption)

 

Economics Data Dashboard (energy demand by fuel user and type)

Economics Data Dashboard (energy demand by fuel user and type)

 

Economics Data Dashboard (employment by industry and region)

Economics Data Dashboard (employment by industry and region)

 

Please note that all the values displayed in the charts, plots and maps within the previous four images of the different dashboards are indicative only.

Google API-powered heatmap viewer of student visitor numbers at Newcastle University Open and Visit Days

As part of the School of Civil Engineering and Geosciences involvement in University student recruitment activities, prospective 6th form and college students can attend Open and Visit Days. These days give students the opportunity to come and learn a little bit more about the courses that are offered at the University, including those taught within the School. Within Geomatics, prospective students are given some experiences of what it might be like to study Geographic Information Science (GIS), or Surveying and Mapping Science (SMS) Undergraduate courses via a handful of taster exercises. These exercises are designed to enable staff members to talk about some of the basic concepts that a prospective student might learn about should they decide to apply and study GIS or SMS.

A key student recruitment activity within the School and more widely the University, involves the coordinated marketing and distribution of promotional materials focussed on Undergraduate courses to different colleges and schools around the UK.  In order to better understand how the School’s involvement in this activity leads to prospective students attending the University Open and Visit Days, thus showing an interest in the courses on offer from the School, a very simple web-based tool has been developed to record where prospective students are travelling from on Visit and Open Days, by recording against the school or college at which the student attends. However not only does this begin to allow recruitment staff to understand how marketing activities are leading to prospective students attending the Visit and Open Days, it also doubles as a taster exercise in explaining some of the basic concepts of data capture, management and visualisation that a student would learn more about within the GIS and SMS courses. A prospective student is able to search for the school or college that they attend from a geocoded set of more than 60,000 schools, and then subsequently increment a count against that particular school for the particular year in which they attended a Visit or Open Day. All this information is stored within a PostGIS-enabled PostgreSQL relational database, and is served out to the webpage via JSON following the use of standard SQL queries to query the underlying data. As a result a prospective student, as well as recruitment staff, are able to create custom Heat Maps (intensity, not temperature!), all powered by the Google Maps API, of their data, or data from previous years. The query interface allows different HeatMaps to be created based on sub-selections of school type, gender (boys only, girls only, or mixed gender schools) and years of interest.

For clarification the database stores no other information about the student other than a count against a particular school or college at which the prospective student attends, and the addition of new information is protected behind a username and password. The following images give some illustrations of this interface and tool:

Increment count against a school, at which a prospective student attends

Increment count against a school, at which a prospective student attends

 

HeatMap viewer, with criteria dialog

HeatMap viewer, with criteria dialog

 

HeatMap viewer, outputs

HeatMap viewer, outputs

ITRC Assembly, June 10th-12th 2013, Chilworth Manor Hotel, Southampton

During the early summer of 2013, the UK Infrastructure Transitions Research Consortium (ITRC) underwent a mid-term review, approximately two and half years after the inception of the research programme, which coincided with the annual ITRC Assembly. The assembly and review gave all of those working within the consortium, and also invited guests and delegates, the opportunity to hear about the work accomplished during the initial half of the research programme. The 5 year research consortium is funded through an EPSRC Programme Grant, with the mid-term review offering the chance to discuss the future of the flexible funding available for the final two and half years of the programme.

The three day meeting was held at Chilworth Manor Hotel in Southampton, and was facilitated by a facilitation group, Dialogue Matters to help coordinate and focus a delegation of researchers, academics, stakeholders and partners. Monday offered the chance for the Expert Advisory Group (EAG) to review the documentation and work completed under the five different work streams. Whilst part of this review took place behind closed doors alongside the program’s principal and co-investigators, subsequently the EAG gave direct feedback to all of those attending the three day meeting. This session was then followed by an open floor discussion and questioning by researchers, PhD students and investigators from the program of the EAG panel. The utility of the EAG in particular was felt by the program during these discussions, and also via their continued guidance on the cycle 2 and 3 assessments due for release in January 2014 and in the autumn of 2015 respectively. Further to the feedback delivered, the post-lunch slot was dedicated to researchers and investigators funded through the program the chance to present some more specifics about the tasks undertaken during the first half of the program. This was particularly effective in getting everyone up to speed with what others within the consortium had been working on, and helped certainly to set the scene for discussions about future directions assigned to day two and three. Finally, a group of ITRC-affiliated PhD students presented some scoping research they had carried out to try to pull together a set of data on projects, research centres and institutes at the global scale who are also working on similar research as that conducted under the ITRC banner. Not only was the presentation interesting in the manner in which it was delivered, the data and information collected offered a great starting point for further development of the ever-growing research community, acting as a focal point for information about the community at large.

Day two began to offer the affiliated researchers and investigated across the many universities represented within the program, the opportunity to address some of the following questions:

  • Where have we got to?
  • What is happening in this field, in other projects and around the world?
  • What externalities may adjust the way this research is conducted, or will influence the likely impact the research has e.g. changes in policy, education, funding, society, environment, markets etc)

Whilst discussions of these questions began immediately in the morning session to broaden the horizon of future possible directions, a selection of “seed” ideas or possible projects that were a priori selected as being potential key research directions were also considered. The opportunities to think more broadly about possible research directions for the final two and half years of the project and also consideration of ideas already identified as of interest, gave everyone the chance to give their opinion on what could or could not feasibly be achieved given the available remaining time and resource. From a personal perspective, I think this gave everyone a real sense of ownership of the future direction of the research and certainly helped to gauge the relative importance of the different tasks identified by researchers from wholly different backgrounds. Subsequently this session allowed researchers to consider new ideas and areas based on the knowledge gained during the first half of the program. The breadth of ideas was enormous, ranging from the need for autonomous analytics for infrastructure planning, provision, monitoring and recovery to the need for new systems to manage the proposed integration of unmanned aerial vehicles within commercially used airspace in the United States, currently being considered by FAA.

Whilst the majority of the second day was spent considering the future direction of program, the afternoon session gave an opportunity for those involved to take stock of the success of the mechanisms employed for internal communication within the consortium. As the consortium is spread over many research centres and universities, effective communication between them and within the consortium is critical to ensuring objectives are achieved. The qualitative review considered the utility of using social media to facilitate communication both internally and externally, such as the use of Twitter and Skype for external dissemination and internal discussions, whilst also appraising the use of the ITRC intranet for collaborative working, and assessing the state of the external facing ITRC website.

With Tuesday giving plenty of opportunity to widen the research agenda and look at possible future research directions that the consortium could move in to, as well as assessing what tasks are to be achieved within the remaining two and half years of the project, Wednesday’s agenda focussed on narrowing this scope. A series of research themes had been identified from Tuesday’s discussions, and researchers were invited to select a theme upon which to discuss what the key areas of interest within that theme might be. However, not only were ideas generated, but challenges to achieving success in these areas were also highlighted, to give an impression of the relative difficulty of each theme. The results of many of the discussions held on day two and three have certainly helped the principal and co-investigators of the program to coordinate what tasks and objectives are to be achieved within the final years of the program.

Overall the assembly and mid-term Review offered everyone involved in the program to take stock of the achievements to date, whilst recognising the significant challenges that lay ahead when trying to deliver on a program which is trying to understand the complex nature of infrastructure, how it is operated, and it’s likely resilience to impending changes in demography, economy and climate.

The following table offers a summary of those people who were involved in the three day meeting:

Role Name Affiliation
ITRC Expert Advisory Group (EAG)

Chairman

Colin Harris Independent
ITRC Expert Advisory Group (EAG)

Members

Rosemary Albinson BP
  Theresa Brown Sandia National Laboratories
  Jeremy Cooper Laing O’Rourke
  Yacov Haimes University of Virginia
  Geoffrey Hewings University of Illinois
  David Penhallurick HM Treasury
  Margot Weijnen TU Delft
     
EPSRC Representatives Christopher White EPSRC
  Iain Larmour EPSRC
     
ITRC Principal Investigator Professor Jim Hall University of Oxford
     
ITRC Program Manager Miriam Mendes University of Oxford
     
ITRC Investigators Dr Nick Eyre University of Oxford
  Professor John Preston University of Southampton
  Professor Chris Kilsby Newcastle University
  Professor William Powrie University of Southampton
  Professor Cliff Jones Newcastle University
  Dr Stuart Barr Newcastle University
  Dr Stephen Hallet Cranfield University
  Professor Pete Tyler University of Cambridge
  Professor Mark Birkin University of Leeds
  Dr Jim Watson University of Sussex
     
ITRC Researchers Simon Abele University of Oxford
  David Alderson Newcastle University
  Pranab Baruah University of Oxford
  Simon Blainey University of Southampton
  Modassar Chaudry Cardiff University
  Adrian Hickford University of Southampton
  Scott Kelly University of Cambridge
  Alexander Otto University of Oxford
  Raghav Pant University of Oxford
  Meysam Qadrdan Cardiff University
  Chris Thoung Cambridge Econometrics
  Rachel Beaven Cambridge Econometrics
  Martino Tran University of Oxford
  Chengchao Zuo University of Leeds
     
ITRC-affiliated PhD students Edward Byers Newcastle University
  Robert Carlsson University of Oxford
  Razgar Ebrahimy Newcastle University
  Timothy Farewell? Cranfield University?
  Ed Oughton University of Cambridge
  Oliver Pritchard Cranfield University
  Scott Thacker University of Oxford
  Katherine Young University of Oxford

ITRC WS1 Visualisation Workshop: Visualisation of multi-dimension data, 22/05/2013, St Hugh’s College, University of Oxford

As the ITRC programme progresses and approaches the mid-term review stage, in June and July of 2013, the of the work stream 1 (WS1) infrastructure capacity and demand modelling teams are beginning to produce outputs from their next round of modelling. Furthermore, the parallel development of spatial infrastructure networks as part of work stream 2 (WS2), is beginning to raise some significant challenges in terms of appropriate and effective data dissemination, communication and interpretation. The underlying high-dimensionality nature of the data being produced as part of WS1 for example, coupled with the complexity of the networks generated as part of WS2 means the consortium as a whole needs to begin to think about appropriate mechanisms to visualise these data.  For example, some initial prototypes of possible visualisation tools are beginning to be developed, (see here), but rather than build and design tools from the perspective of one researcher, it was considered more appropriate to consult with, other similar projects who are visualising similar data, or will require the ability to visualise similar data in similar ways to that required of ITRC, and also a host of visualisation and design experts from around the UK to gain better perspectives.

An initial workshop, organised by ITRC members, Dr Alex Otto (ITRC WS1 investigator), Dr Greg McInerny (Senior Research Fellow, University of Oxford), Mr David Alderson (Researcher in GeoInformatics, Newcastle University), Dr Stuart Barr (Senior Lecturer in Geographic Information Science, Newcastle University) and Miriam Mendes (ITRC Programme Manager, University of Oxford), sought to bring together relevant researchers from the plethora of Adaptation and Resilience in a Changing Climate (ARCC) network projects and leading researchers and experts in the field of data visualisation and design. Prior to the workshop, a questionnaire was distributed to both the invited ARCC project representatives and the visualisation experts in an attempt to give the organising team a better centralised perspective of what the respective groups would want to hope to gain by attending the workshop. The responses were then studied to tease out any overlaps between visualisation challenges faced across the ARCC projects, to attempt to collate a set of discussion points upon which to focus discussions in the afternoon of the workshop. Prior to these more focussed discussion sessions, the workshop initially allowed the ARCC project representatives to briefly (in 5 minutes or less) explain the nature of the project in which they are working, but also describe and explain some of the visualisation challenges being faced within that project. The aim of this early session was to allow the visualisation experts time to understand the background of the projects themselves, and also the nature of some of the data being produced, such that the more focussed discussions taking place in the afternoon had a little context.

From the responses to the questionnaire, and also following the morning’s ARCC project overview session, a series of 5 discussion topics were devised, that attempted to encapsulate the common visualisation challenges across all the projects, and are listed below.

  • Visualising multiple dimensions and scenarios;
    • Chair: Martino Tran (ITRC – University of Oxford)
    • Rapporteur: Craig Mills (Visualisation – UN)
  • The spatial dynamics of infrastructure networks;
    • Chair: Scott Thacker (ITRC – University of Oxford)
    • Rapporteur: Martin Austwick (Visualisation – UCL)
  • Temporal visualisation of infrastructure behaviour and response;
    • Chair: Sean Wilkinson (RESNET – Newcastle University)
    • Rapporteur: Min Chen (Visualisation – University of Oxford)
  • Simplifying and communicating effectively complex model outputs;
    • Chair: Jason Dykes (Visualisation – City University, London)
    • Rapporteur: Scott Kelly (ITRC – Cambridge University)
  • Multi-disciplinary co-production for infrastructure visualisation.
    • Chair: Simon Blainey (ITRC – University of Southampton)
    • Rapporteur: Jane Lewis (Reading e-Science Centre, University of Reading)

A chair and rapporteur, selected from the list of workshop attendees was devised such that each topic had a representative from the ARCC network, and from the visualisation community. Each topic was then discussed by attendees for about 10 minutes, with the chairs and rapporteurs capturing the salient points discussed around that particular topic. After 10 minutes of discussion the attendees subsequently moved on to the next discussion topic and a different table. Overall as a format for delivering break out sessions, this quick-fire, round-robin approach seemed to work well, allowing all attendees to discuss all the common discussion topics about visualisation, whilst at the same time having the discussions steered and reported by representation from both sides. The approach also seemed to help stimulate discussions between project representatives and visualisation experts, which was one of the objectives or organising and delivering the workshop. However further work is currently being undertaken to transform some of the excellent discussions in to a positioning paper with respect to visualising high dimensionality data for infrastructure planning and provision purposes. It is hoped that representatives from the projects, particularly those organising the workshop and on the ITRC side will be looking to further engage and collaborate with the visualisation community.  Watch this space…

Links to presentations split by those relevant to different communities are listed below:

Full Attendee List

ARCC Project-affiliated attendees (* speaker on visualisation challenges)

ARCC Project Representative Affiliation
ITRC Alex Otto* University of Oxford
ITRC Stuart Barr* Newcastle University
ITRC David Alderson Newcastle University
ITRC Raghav Pant University of Oxford
ITRC Scott Thacker University of Oxford
ITRC Jim Hall* University of Oxford (Principal Investigator – ITRC)
ITRC Miriam Mendes University of Oxford (Programme Manager – ITRC)
ITRC Simon Abele University of Oxford
ITRC Alex Leathard Newcastle University
ITRC Meysam Qadrdan Cardiff University
ITRC Modassar Chaudry Cardiff University
ITRC Simon Blainey University of Southampton
ITRC Kate Young University of Oxford
Transport Utilities’ Conversion Points (TUCP) Liz Varga* Cranfield University
All-in-One Tomasz Janus De Montfort University, Leicester
Undermining Infrastructure Jonathan Busch* University of Leeds
Land of the MUSCos Christof Knoeri* University of Leeds
Step-change Miles Tight* University of Birmingham
RESNET Sean Wilkinson* Newcastle University

 

Visualisation / Design Experts (presentations and speakers listed below):

Greg McInerny University of Oxford, Microsoft Research
Min Chen University of Oxford
Craig Mills United Nation Environmental Program World Conservation Monitoring Centre
Jason Dykes City University, London
Jane Lewis Reading e-Science Centre, University of Reading

 

Other invited attendees:

Vicky Hayman UK Climate Impact Projections, University of Oxford
Chris Cooper IBM, London
David Miller IBM, London
Mathew Carlos University of Oxford
Zoe Austin University of York
Martin Austwick UCL
Craig Robson Newcastle University
Glenn Hart Ordnance Survey
Paula Engelbrecht Ordnance Survey
Andrew Munslow Met Office

 

UK Infrastructure Transitions Research Consortium (UK ITRC) University of Oxford Newcastle University

Linking OpenLayers, D3, JSON and NetworkX to build a topological and geographic network viewer

As many of the networks that I am building as part of my involvement in the Infrastructure Transitions Research Consortium (ITRC – www.itrc.org.uk) are inherently spatial, I began thinking about how it might be useful to be able to visualise a network using the underlying geography but also as an alternative, the underlying topology. I began exploring various tools, and libraries and just started playing around with D3 (d3js.org).  D3 is a javascript library that offers a wealth of widgets and out-of-the-box visualisations for all sorts of purposes. The gallery for D3 can be found here. As part of these out-of-the-box visualisations it is possible to create force directed layouts for network visualisation. At this stage I began to think about I can get my network data, created by using some custom Python modules, nx_pg and nx_pgnet and subsequently stored within a custom database schema within PostGIS (see previous post here for more details), in a format that D3 can cope with. The easiest solution was to export a network to JSON format as the nx_pgnet Python modules allow a user to export a networkx network to JSON format (NOTE: the following tables labelled “ratp_integrated_rail_stations” and “ratp_integrated_rail_routes_split” were created as ESRI Shapefiles and then read in to PostGIS using the “PostGIS Shapefile and DBF Loader”).

Example Code:

import os

import osgeo.ogr as ogr

import sys

import networkx as nx

from libs.nx_pgnet import nx_pg

from libs.nx_pgnet import nx_pgnet

conn = ogr.Open(“PG: host=’localhost’ dbname=’ratp_networks’ user='<a_user>’ password='<a_password>'”)

 #name of rail station table for nodes and edges

int_rail_node_layer = ‘ratp_integrated_rail_stations’

int_rail_edge_layer = ‘ratp_integrated_rail_routes_split’

 #read data from tables and create networkx-compatible network (ratp_intergrated_rail_network)

ratp_integrated_rail_network = nx_pg.read_pg(conn, int_rail_edge_layer, nodetable=int_rail_node_layer, directed=False, geometry_precision=9)

 #return some information about the network

print nx.info(ratp_integrated_rail_network)

 #write the network to network schema in PostGIS database

nx_pgnet.write(conn).pgnet(ratp_integrated_rail_network, ‘RATP_RAIL’, 4326, overwrite=True, directed=False, multigraph=False)

 #export network to json

nx_pgnet.export_graph(conn).export_to_json(ratp_integrated_rail_network, ‘<folder_path>’, ‘ratp_network’)

 

Having exported the network to JSON format (original data sourced from http://data.ratp.fr/fr/les-donnees.html), this was then used as a basic example to begin to develop an interface using D3 to visualise the topological aspects, and OpenLayers to visualise the spatial aspects of the network. A simple starting point was to create a basic javascript file that contained lists of networks that can be selected within the interface and subsequently viewed. Not only did this include a link to the underlying file that contained the network data, but also references to styles that can be applied to the topological or geographic views of the networks. A series of separate styles using the underlying style regimes of D3 and OpenLayers were developed such that a style selectable in the topological view used exactly the same values for colours, stroke widths, fill colours as styles applicable in the geographic view.  These stylesheets, stored within separate javascript files are pulled in via an AJAX call using jQuery to the webpage, subsequently allowing a user to select them. Any numeric attributes or values attached at the node or edge level of each network could also subsequently be used as parameters to visualise the nodes or edges in the networks in either view e.g. change edge thickness, or node size, for example. Furthermore, any attribute at the node or edge level could be used for label values, and these various options are presented via a set of simple drop down menu controls on the right hand side of the screen. As you may expect, when a user is interested in the topological view, then only the topological style and label controls are displayed, and vice versa for the geographic view.

For spatial networks, the geographic aspects of the data are read from a “WKT” attribute attached to each node and edge, via a WKT format reader to create vector features within an OpenLayers Vector Layer. It is likely this will be extended such that networks can be loaded directly from those being served via WMS, such as through Geoserver, rather than loading many vector features on the client. However for the purposes of exploring this idea, all nodes and edges within the interface on the geographic view can be considered as vector features. The NodeID, Node_F_ID, and Node_T_ID values attached to each node, or edge respectively as a result of storing the data within the custom database schema, are used to define the network data within D3.

At this stage it is possible to view the topological or geographic aspects of the network within a single browser pane. Furthermore, if graph metrics have been calculated against a particular network and are attached at either the whole graph, or individual node or edge level, they too can be viewed within the interface via a series of tabs found towards the bottom. The following image represents an example of visualising the afore-mentioned Paris Rail network using the interface, where we can see that the controls mentioned, and how the same styles for the topological and geographic views are making it easier to understand where one node or edge resides within the two views. The next stage is to develop fully-linked views of a network such that selections made in one window are maintained within another. This type of tool can be particularly useful for finding disconnected edges via the topological view, and then finding out where that disconnected edge may exist in it’s true spatial location.

Example of the geographic view of the Paris Rail Network displayed using OpenLayers (data read in from JSON objects, with geometry as WKT)

 

Example of graph metrics (degree histogram) for Paris Rail Network (data stored at network level)

 

Example of the topological view of the Paris Rail Network displayed using force directed layouts from D3.js