Jul 162015

One of our key aims in building the interface for our collection was to allow people to explore and “play with” the data. It’s hard to get a sense of the extent of the series and the relationships between the surveys without some kind of overview: once you can see the surveys all together and look at them in different ways, it’s much easier to grasp their logic. So we wanted a tool that would aggregate all of the information we have gathered and then allow people to look at that information in flexible ways, to filter and explore it according to their interests.

Flexibility was also a priority in technical terms: we’re making this data available for the first time in this format, so we are aware that we don’t really know what people will want to do with it. We don’t see what we have done with the demonstrator as being the last word but rather the first. Based on this, we can start to understand the data better and start to understand how people might want to access it.  We expect to have to adapt the data and the ways of accessing it as we go along and we learn what we can most usefully provide to the community.

The Data

The process of gathering data has been described in another post, but from the demonstrator’s point of view what was important was to try to keep things as general and adaptable as possible. Nevertheless, this kind of historical data presents certain peculiarities and challenges. One of the most obvious is how to present the survey data. The surveys are arranged by county but the counties that were used are not the counties as they are today. Indeed, the counties used in the first and second phases of the county surveys are not the same. So we needed a mechanism which would allow people to make sense of the data without being restrictive. We’ve achieved this by providing a canonical list of counties taken from Ordinance Survey Data from the early 19th century. We then map this to the actual counties as surveyed. There’s not a perfect match here but we take a “permissive” view of the data – we’d rather show you slightly too much than too little. So the user gets presented with the canonical list in the search facility and we then map that to the county data to decide what to show. The same holds for the author data. We hold a canonical list of authors and map these to the real authors. This allows us to adjust the data in future as we discover more about it.

The Data Model

This mapping then gives rise to the data model. We have surveys which have a county associated with them. Then we have a list of counties which we present to the user which may map to more than one of the underlying counties. That can get a bit confusing but if we look at an example, it becomes clear. If we want to look at the surveys for Shetland then in the filter list we have “Zetland or Shetland” which is how it is listed in the Ordinance Surveys. In the first phase of the surveys, Shetland was included under “Northern Counties and Islands” but in the second phase it has a survey of its own. The implication of this for the data model is that we have to have a one-to-many mapping from entries in the search list to the entries in the surveys. In fact, the same county survey might appear under more than one search term e.g. the first phase “Northern Counties and Islands” needs to appear under Shetland, Orkney, Caithness and Sutherland. So we have to have a many-to-many mapping between the search counties in the interface and the counties as specified in the surveys themselves. To do this we adopt the standard database approach of having a mapping table i.e.ccounty_county

So ccounty is the list of counties as it appears in the search list and county is as they appear in the surveys and the mapping table allows us to relate these two to each other in any way we want. Each Survey can have many publications and each publication can be held in multiple places. This explains why we have separated out surveys from publications from holdings in the data model.

database schema

Database Schema (click to open in new tab)

This model might seem a little complex but it gives us a great deal of flexibility in how we handle counties and authors and makes it fairly easy to add new information about publications and holdings as it becomes available to us.

The Technology Chosen

In line with the ethos of flexibility, we decided to work with standard technology components. At the back end is a relational database. Sitting on top of that is a Web Application built using a standard MVC framework. This approach has advantages in terms of the flexibility but also in terms of getting up and running quickly. The MVC approach (Model-View-Controller) separates out the storage of the data (the Model) from the logic of the application (the Controller) and how the data is displayed (the View). This means that changing one part of it has less impact because it is isolated from the other components. A good example of this flexibility is the change we made to the interface which was covered in a previous post.

The MVC approach to web applications is one of the standard development techniques for web applications these days and when it comes to implementing this you have a wide choice of languages and MVC systems. In our case, it’s all written in Perl using Postgres for the DB with a Catalyst Application on top. So the application takes the standard Catalyst approach of using DBIx::Class to implement the Model and interface to the database and Template Toolkit for the front end. The choice of specific MVC implementation doesn’t matter so much – there are plenty to choose from! It’s really the flexibility this approach gives which is the main thing. Using standard technologies gives us the adaptability we need to be able to do this easily, so that we can get the data available and we can adapt to whatever changes come out of that down the line.

Evolution by Use

So this demonstrator gives people access to look at the data. We’re hoping people will find it helpful in “playing with” the data. But it’s very much the first draft. We expect it to evolve over time as we and any one else interested in the Surveys gets to know the data better and we start to understand more about how to make this data available to people.

Apr 292015

1893 map of Shetland, from Cassell’s Gazetteer of Great Britain and Ireland; Published by Cassell and Company Limited, London.

One of the aims of our current project is to establish the cost and workflow requirements for creating a complete virtual collection of the County Surveys.  Many of the surveys are already available in various online archives but discovering them is not as easy as it could be and quality and accessibility remain quite variable. In the longer term, we hope to aggregate high quality full-text files that we can use for research-led text mining.  In order to establish the projected costs and labour involved in such a project, as part of the pilot we plan to identify one or two rare surveys and digitise them according to current best practices, documenting this process for ourselves and others. Clearly, as funds are limited, it makes sense to focus on volumes that are not already available in digital form and which are rare even in print.

One such candidate is the General view of the Agriculture of the Shetland Islands by John Shirreff which was published in Edinburgh by Constable & Co in 1814. This is a volume, according to one early 19th Century reviewer, which was of a peculiarly special interest to contemporary readers for it describes “a remote part of the British dominions, with which many readers are perhaps as little acquainted as with the Islands in the South Sea; and they exhibit a state of Society very different in several respects from that which prevails in the other provinces of Britain.”  Indeed, comparing Orkney and Shetland to the wilds of the American frontier, he suggests the inhabitants of these northern islands belong to a different, less civilised time and “bring into view a stage in the progress of improvement at which the inhabitants of the South has arrived some centuries ago, and which had been long since passed over by the people of almost every other part of the Island.” (The Farmer’s Magazine 15 (Aug 1814): 343) The exoticism, snobbery and geo-political bias of these remarks seems almost comical today, but they suggest that the contents of Shetland survey may be of particular importance to historians given the apparently substantial differences from more ‘advanced’ mainland practices.  Happily we will all be able to judge for ourselves soon, because a print copy of the Shetland survey is held here in Edinburgh at the Royal Botanic Gardens and they have kindly agreed to allow its digitisation: we’ll post about this process once it gets underway.”

Mar 312015
Folded Map from the 'revised' survey of North Wales published in 1810

Folded Map from the ‘revised’ survey of North Wales published in 1810

As I described in an earlier post, cross-checking between bibliographies and catalogues has formed a substantial part of the work that has so far gone into building our bibliographic database of the holdings of the County Surveys.  In order to make sure that we are identifying and locating as many of the surveys as we can, we have also been checking our holdings information against county maps. If we can say that we have found surveys that document all the counties, then we can be confident that we have the significant majority, if not a complete set.

The task is complicated by the history of the counties in Britain. Firstly, the borders of regions, districts and other administrative areas have changed frequently over the last two hundred years, meaning that the areas referred to by a county name in 1800 can be different to the areas referred to by the same name now. Secondly, as these changes have taken place, the names used to refer to areas have also changed, in some cases quite dramatically. What we now call Dumfries and Galloway, for example, was historically three counties, Dumfriesshire, Wigtownshire and ‘the Stewartry of Kirkcudbright’. We decided that for the twenty-first century researchers who would be consulting our online collection, the ability to search by modern names would be important.

To map our list of survey holdings to the geography of the British Isles and to the areas defined by modern names, we turned to the Ordnance Surveys. Ordnance Survey have been responsible for mapping and surveying the UK since the 1790s, the same decade in which the County Surveys were commissioned. Their most recent county maps represent Britain in the mid-Twentieth Century, when counties were still the primary administrative entities across the nation (since the 1970s, regional authorities have replaced counties), so the list of their maps represented as authoritative a list of modern counties as we are likely to find.  This has become both our ‘canonical county name list’ and the list of geographical areas against which we can map the areas covered by the surveys.

We hope, in time, that this work will enable us to create an intuitive map interface for our collection.  In itself, though, the process of mapping the surveys to the areas has been very valuable in that it has turned up quite a few queries about our holdings data and about the surveys themselves. For example, it has enabled us to identify no less than four surveys that deal with the same geographical area under different titles, for example Kings County in Ireland is also surveyed as Offaly, and the ‘Central Highlands’ survey of series one describes itself as dealing with Perthshire, although there are three other surveys with Perth in the title published in the same series.  The process has also allowed us to check geographically neighbouring surveys in order to establish the boundaries of the areas dealt with: for instance, in the first series there appears to be no survey for Bute. Checking Clydesdale, Argyle, and Ayr, the areas which might conceivably contain or neighbour Bute, reveals no trace of the Isle, suggesting that it was probably completely neglected during the first phase rather than incorporated in another survey.

This kind of cross-checking and mapping is time-consuming but necessary to ensure the integrity of the data and of our mode of presentation. In a sense it is an echo of the work done by the original surveyors, who also sought to compile information about areas and fields for which there was no guide place. “In obtaining an account of the present state of husbandry in North Wales several difficulties occurred”, wrote George Kay the first surveyor of the area in 1794. “Among others, no distinct map of it could be procured, although I enquired at all the shops in the principle towns from Edinburgh to Chester.” (p.1) Charting previously uncharted territories, the surveyors laid the foundations of Sir John’s pyramid of enquiries, enabling new questions to be posed and answered. At a much more modest scale, we hope our online collection will help researchers chart the bibliographic landscape of the Surveys and facilitate new research on this fascinating material.

Jan 222015

Cover of: The code of agriculture by Sinclair, John Sir

The County Surveys project is closely related to EDINA’s Statistical Accounts of Scotland service, for together the surveys and the accounts form the foundations of Sir John Sinclair’s ambitious ‘pyramid of agricultural enquiries’. The insights and lessons gleaned from these extensive reports were collected in Sinclair’s 1817 book The Code of Agriculture, the pinnacle of his pyramid, which represented the culmination of decades of work with the aim of ‘rendering, a general knowledge of the principles of husbandry, more easily accessible’.

Although they both belong to one larger vision however, there are significant differences between the Statistical Accounts and the County Surveys. Most obviously, they focus on different geographical entities. The Statistical Accounts are concerned with individual parishes and are confined to Scotland, while the Surveys broadened the scope of the enquiries across Britain and Ireland and considered farming practices and land management at the level of county. They therefore provide information about much greater and more diverse areas, enabling comparison across different regions and nations. In this lay their value: the aim of the enquiries was to excite the spirit of ‘improvement’ across the land and, by comparing practices, farmers and landowners could pool their knowledge and share the benefits of others’ experience. Some of the surveys were published with appendixes encouraging such cross-pollination of ideas and methods, for example the General View of the Agriculture of Northampton contained a comparison of ‘the English and Scotch Systems of Husbandry, as Practised in the Counties of Northampton and Perth.’

Another significant difference between the parish accounts and the surveys lay in their authorship. The accounts were written by local ministers: living among the communities they described, they knew intimately the circumstances of their flock but were often less  informed about the other subjects on which they were tasked to report. This led to some far-fetched speculations and assertions, such as the fantastic tale of a (non-existent) loch, reportedly 100 fathoms deep, at the summit of Meallfuarvonie, a hill in Glen Urquhart.  As David McVey puts it,  ‘simple facts of topography and natural history ,[…] sometimes wander into the realms of folklore in the [Old Statistical Accounts] – a warning, perhaps, to approach other information with some caution.’ And, as McVey goes on to point out,

‘sometimes what is omitted is more telling than what is included. In the entry for the Parish of Golspie, scene of some of the most notorious incidents in the Highland Clearances during the early 19th century, the Revd Alexander MacPherson made no mention of the resulting hardship, merely commenting that there had been a decline in the parish’s population occasioned by agricultural improvements. He did, however, indulge in several purple passages of praise for the House of Sutherland, the sole landowners in the parish.’*

These biases, inventions, and idiosyncrasies are part of what makes the accounts such rich resources for historical research.

In contrast, the surveys were commissioned from laymen rather than clergy, sometimes surveyors but often ‘intellectual farmers’ and others who had a professional interest in agriculture. If they are less idiosyncratic, however, their authorship is no less revealing in terms of prevailing attitudes and social and moral codes. Consider, for example,  William Aiton’s impassioned attack on the country ‘sports’ of Ayrshire.  Describing ‘low jockies’, who beat their charges to perform, Aiton writes:

I consider it to be unworthy of the noblemen and gentlemen, who patronise these races, to give their money to such men, and for such purposes. […] The races at country towns are still worse and ought to be prohibited. […] The exhibition is calculated to shock the feelings of humanity. Three or four meagre, jaded, lame, starved animals, are mounted upon, by as many unfeeling blackguards in a state of intoxication, galloped on a rough, hard road, for several miles and cruelly cut and bloodied with whip and spur, for the amusement or sport of the rabble that attend. Such exhibitions are unworthy of an enlightened age, shocking to humanity, and none can be innocent who contribute to, or give them countenance.  (William Aiton, General View of the Agriculture of the County of Ayr  p.577)

Aiton’s remarks are illuminating in several respects, highlighting attitudes towards animals, towards the nobility and the poor. They show that the surveyors brought their own opinions and concerns to their reports, creating thick, rich descriptions that offer valuable insights into rural life in the late 18th century.

The Statistical Accounts and the County Surveys are ‘sister’ enquiries rather than twins. They complement and enrich one another, enabling researchers to compare different perspectives at different scales, and local concerns and practices at slightly different historical moments.  Creating an online collection of the County Surveys, a sustainable service like the Statistical Accounts of Scotland, would be a step towards digitally reconstructing Sir John’s pyramid and would enable researchers to easily read across and between both sets of documents, enabling new insights and readings of both.


* McVey, David. ”The Scottish “Domesday”‘ Practical Family History, Sept 2010, pp. 46-47)