Post mortem on 26 jul 2016 incident
On July 26th 2016, the following incident occured that lead to the loss of data:
In an attempt to improve FlexLists and take responsibility for it's quality, we outlined responsible disclosure conditions and a hall of fame for security researchers. This program lead to several security issues and other bugs to be fixed over the past few months.
As part of this program an automated scanning tool was used by one of the researchers that triggered an unforseen SQL injection bug. This bug lead to the updating of all column names and types across all lists instead of a single one.
As soon as this coruption was discovered we fixed the cause of the bug and made an attempt to recover the data. We indeed were able to quickly restore a copy of our database. However, at that time, the altered data had already ended up in our remote backup. This due to the backup running in the middle of the night, only an hour after the data was altered.
The loss of data was caused by the combination of two oversights: sql injection and backup strategy.
The root of the SQL injection vulnerability was in code that originated from our original 2006 prototype. We since protected the code against this vulnerability, however at this particular instance an assert statement was missed. In hindsight there are two reasons this particular instance was missed:
- Reviewing the code on it's own, the data coming in seemed like it would be used with sanitized data only and only for internal use
- There was a similarly named function doing similar things that did have all the proper fixes, so at the original time the vulnerability was fixed, the second method was overlooked
The second part was our backup strategy. At the time, running at a VPS on a fully redundant platform and having a daily remote copy of the full server at an independent provider felt like a proper way to protect against data loss as well as admin failure. However, we did not consider a scenario where data would be corrupted and continue to be backed up. This was ofcourse a huge oversight.
We are very sorry for this loss of data and offer our most sincere appologies to everyone affected by this.
Actions we have taken since the incident:
- The exposed vulnerability was fixed
- A weekly and monthly backup copy was added to our strategy to have a better chance at recovering recent data.
- Existing code was reviewed again for similar vulnerabilities
- We recovered column names for lists created before 2009
- We partially recovered column types where we were able to determine the proper type from the data
Changes that we will implement in the near future
- Old code touching the database will be replaced with code that structurally prevents SQL injection (PDO with prepared statements)
- Old and stable code needs to be revisited regularly to assert security
- Separating out old code into objects/functions can hide security issues
- Backup strategies do no longer need to protect only against system/human failure, but also against intentional data curruption