BCS SPA2016

SPA Conference session: Working fearlessly with your data

One-line description:A session exploring ways we can minimise risk while make big changes to the way our data is stored
 
Session format: Case study / goldfish bowl [read about the different session types]
 
Abstract:We often talk about being able to change our code fearlessly, but many teams still avoid changing the way their production data is stored, or 'refactoring their data', because they fear corrupting it in a way that will be hard to recover from.

In this session we'll talk about a case study from our work at spabreaks.com where we migrated millions of attachments from a CouchDB database to Google Cloud Storage without our users noticing. We'll cover how minimised the risk to our data and load on our production systems by running both storage systems in parallel, and how we built a proxy so that the application manipulating this data was unaware of the changes in the way it was stored.

After we've presented the case study, we'll lead a discussion around the more general techniques that can be used when making big changes to our production data, such as parallel running, monitoring and instrumentation, 'migrate on write' & log-structured storage systems.
 
Audience background:Some prior experience of working on software that talks to a database (of pretty much any kind) would be useful, and we hope that some people will have experiences of working on large-scale data migrations in production.
 
Benefits of participating:We hope that participants will come away with some practical examples of how they can work more safely with their data in production.
 
Materials provided:Presentation slides
 
Process:- a presentation of our specific case study
- a presentation about how the techniques we used can be generalised
- a goldfish bowl discussion about the experiences of others in implementing these techniques and problems that other teams have had in migrating data in the past
 
Detailed timetable:0:00-0:30 Presentation of our case study: the problem we had, and how we solved it
0:30-0:45 Presentation about generalising the techniques we used
0:45-0:75 Goldfish bowl discussion
 
Outputs:Emailed summary of the discussion for participants: did any new techniques come up? What kind of problems are others tackling with migrating their data in production? How do you do this kind of thing on a tight deadline (e.g. when a storage system is running out of space, or there are performance problems in production)?
 
History:None, although I had a brief discussion around it at the London Continuous Delivery meetup.
 
Presenters
1. Stephen Lewis
Spabreaks.com / Yourgolftravel.com
2. Ian Bell 3.