Building data liberation infrastructure – Export, access your data effortlessly

Just as a reminder: the idea of the data mirror is having personal data continuously/periodically synchronized to the file system, and having programmatic access to it.

It might not be that hard to achieve for one particular data source, but when you want to useten or more, each of which with its own quirks it becomes quite painful to implement and maintain over time. While there are many reasons to make it simple, generic, reliable and flexible at the same time, it is not an easy goal.

The main principles of my design are modularity, separation of concerns and keeping things as simple as possible. This allows making it easy to hook onto any layer to allow for different ways of using the data.

Most of my pipelines for data liberation consist of the following layers

• : knows how to get your data from the silos

  The purpose of the export layer is to reliably fetch and serialize raw data on your disk. It roughly corresponds to the concept of the'data mirror app'.

  Export scripts deal with the tedious business of authorization, pagination, being tolerant of network errors, etc.

  Example: the export layer for Endomondo data is simply fetching exercise data from the API (using existing library bindings) and prints the JSON out. That's all it does.

  In theory, this layer is the only essential one; merely having raw data on your disk enables you to use other tools to explore and analyze your data. However, long term you'll find yourself doing the same manipulations all over again, which is why we also need:

• data access layer (DAL): knows how to read your data

  For brevity, I'll refer to it as DAL (Data Abstraction/Access Layer).

  The purpose of DAL is simply to deserialize whatever the export script dumped and provide minimalistic data bindings. It shouldn't worry about tokens, network errors, etc., once you have your data on the disk DAL should be able to handle it even when you're offline.

  It's not meant to be too high level; otherwise, you might lose the generality and restrict the bindings in such ways that they leave some users out.

  I think it's very reasonable to keep both the export and DAL code close as you don't want serializing and deserializing to go out of sync, so that's what I'm doing in my export tools.

  Example: DAL for Facebook Messenger knows how to read messages from the database on your disk, access certain fields (e.g. message body) and how to handle obscure details like converting timestamps to datetime objects.

  • it's not trying to get messages from Facebook, which makes it way faster and more reliable to interact with data
  • it's not trying to do anything fancy beyond providing access to the data, which allows keeping it simple and resilient

• downstream data consumers

  You could also count it as the third layer, although the boundaries are not very well defined at this stage.

  As an input it takes abstract (i.e. non-raw) data from the DAL and actually does interesting things with it: analysis, visualizations, interactions across different data sources, etc.

  For me, it's manifested asa Python package. I can simply import it in any Python script, and it knows how to read and access any of my data.

Next, I'm going to elaborate on implementing the export layer.

Sometimes a service doesn't offer an API. But from the service developer's perspective, it's still very reasonable to have one if you've got backend/frontend communication.

So chances are the service just isn't exposing it, but you can spy on the token/cookies in your browser devtools and use them to access the API.

You can read more about handling such data sources here:

• Reverse engineering APIs using Chrome Developer Tools : an extremely comprehensive and beginner-friendly tutorial
• "How I used Python to find interesting people to follow on Medium" : an example of reverse engineering Medium API and using devtools
• Starbucks should really make their API public : demo of reverse engineering Starbucks Android app, featuring using proxy and forging the signature

Some examples:

• for exporting Messenger data , I'm using fbchat library. It works by tricking Facebook into believing it's a browser and interacting with private API.
• even though Pocket has an API, to get highlights from it you need to spy on the API key they use in the web app

Sometimes a service doesn't offer an API, doesn't use it even internally and serves HTML pages directly instead. Or, reverse engineering the API is so painful scraping becomes a more attractive option.

In addition to the same difficulties you would experience during API exports, there are some extra caveats here:

• authorization is harder: you definitely need username/password and potentially even a 2FA token
• DDOS protection: captchas, Cloudflare, etc.
• or even deliberate anti-scraping measures

For Python the holy grail of scraping is scrapy :

• Advanced Web Scraping Tutorial : bypassing "403 Forbidden", captchas, and more
• self-contained minimum example script to run scrapy

I'm pretty sure there are similar libraries for other languages, perhaps you could start with awesome-web-scraping repo or Ask HN: What are best tools for web scraping? .

For dealing with authorization, my personal experience is that using a persistent profile directory in Selenium is sufficient in most cases: you can login once manually and, reuse the profile in your scripts.

Examples:

• even though Hackernews has an API for public data , there is no way of getting your upvotes/saves without scraping HTML.
• Amazon and Paypal have to be scraped if you want your data.
• my bank, HSBC doesn't have an API. Not that I expected it from HSBC, I don't live in a fairy tale; but even their manual transactions exports are in PDF which I have to parse .

manual export (GDPR/takeout)

I don't have an iPhone, so will only be referring to Android in this section, but I'd imagine the situation is similar.

These days, a service might not even offer a desktop version at all and considering that scraping data off mobile apps is way harder getting it from the phone directly might be an easier option. The data is often kept as an sqlite database which in many ways is even more convenient than an API!

On Android the story is simple: apps keep their data in /data/data/ directory, which is not accessible unless you root your phone. These days, with magisk it's considerably easier; however, it's still definitely not something a typical Android user would be able to do. Rooting your phone can bring all sorts of trouble by triggering root detection (e.g. common in banking apps), so be careful. And of course, phones come unrooted for a reason, so do it at your own risk.

Once you have root you can write a script to copy necessary files from /data/data/ to your target directory, synchronized with your computer (e.g. via Dropbox or Syncthing ).

Examples:

• you can export Whatsapp data by copying /data/data/com.whatsapp/databases/msgstore.db
• scripts for exporting mobile Chrome/Firefox browsing history
• exporting Bluemaestro environment sensor data

Here I am referring to standalone specific-purpose gadgets like sleep trackers, e-ink readers, etc. The distinguishing thing is the device doesn't have Internet access or doesn't talk to any API.

You've got some options here:

• the device is capable of synchronizing with your phone (e.g. via Bluetooth)

  It's probably easiest to rely onhere. If the sync has to be triggered manually, you can benefit from some UI automation .

• the device is running Linux and has Internet access

  That's often the case with e-ink readers.

  You can potentially run the export script on the device itself and send the data somewhere else. Another option is running an SSH server on the device and pulling data from it, but it's quite extreme.

• the device can mount to a computer

  Then, you can use udev to trigger export when the device is plugged in. If udev feels too complicated for you, even a cron script running every minute might be enough.

Examples:

• using kobuddy for semiautomatic exports from Kobo e-ink reader

Every time you want your data, go exhaustively through all the endpoints and fetch the data. The result is some sort of JSON file (reflecting the complete state of your data) which you can save to disk.

incremental export

Map: export layer .

No matter whichways you have to use to export your data, there are some common difficulties, hence patterns that I'm going to explore in this section.

Just a quick reminder ofthe problems that we're dealing with:

• authorization: how to log in?
• pagination: how to query the data correctly?
• consistency: how to make sure we assemble the full view of data correctly without running into concurrency issues?
• rate limits: how to respect the service's policies and avoid getting banned?
• error handling: how to be defensive enough without making the code too complicated?

My guiding principle is: during the export, do the absolute minimum work required to reliably get raw data on your disk. This is kind of vague (perhaps even obvious), so I will try to elaborate on what I mean by that.

This section doesn't cover the exact details, it's more of a collection of tips for minimizing the work and boilerplate. If you are interested in reading the code, here are some of the export scripts and tools I've implemented.

use existing bindings

don't mess with the raw data

don't be too defensive

• never silently fallback on default values in case of errors, unless you're really certain about what you're doing

• don't add retry logic just in case

  In my experience, it's fair to assume that if the export failed, it's a random server-side glitch and not worth fine-tuning - it's easier to simply start the export all over again. I'm not dealing with that in the individual export scripts at all, and using, to retry exports automatically.

  If you know what you're doing (e.g. some endpoint is notoriously flaky) and do need retries, I recommend using an existing library that handles that like backoff .

allow reading credentials from a file

• you don't want them in your shell history or in crontabs

• keeping them in a file can potentially allow for fine access control

  E.g. with Unix permissions you could only allow certain scripts to read secrets. Note that I'm not a security expert and would be interested to know if there are better solutions to that

  Personally, I found it so boilerplaty I extracted this logic to a separate helper module . You can find an example here .

Map: filesystem .

As I mentioned, for the most part I'm just keeping the raw API data. For storage I'm just using the filesystem; all exports are kept or symlinked in the same directory ( /exports ) for ease of access:

find /exports/ | sort | head -n 20 | tail -n 7

/exports/feedbin
/exports/feedly
/exports/firefox-history
/exports/fitbit
/exports/github
/exports/github-events
/exports/goodreads

naming and timestamping

ls /exports/instapaper/ | tail -n 5

instapaper_20200101T000005Z.json
instapaper_20200101T040004Z.json
instapaper_20200101T080010Z.json
instapaper_20200101T120005Z.json
instapaper_20200101T160011Z.json

synchronizing between computers

I synchronize/replicate it across my computers with Syncthing, also used Dropbox in the past.

disk space concerns

Map: data access layer .

As I, all that DAL does is maps raw data (saved on the disk by the) onto abstract objects making it easier to work with in your programs. "Layer" sounds a bit intimidating and enterprisy but usually it's just a single short script.

It's meant to deal with data cleanup, normalization, etc. Doing this at runtime rather than during the export makes it easier to work around data issues, allows experimentation, and is more forgiving if you make some bugs.

As I mentioned in the, I'm trying to keep data retrieval code and data access code separate since they serve very different purposes and deal with very different errors.

Just as a reminder what we get as a result:

• resilience

  Accessing and working with data on your disk is considerably easier and faster than using APIs.

• offline

  You only access data on your disk, which makes you completely independent on the Internet.

• modularity and decoupling: you can use separate tools (even written in different programming languages) for retrieving and accessing data

  That's very important, so we all can benefit from existing code and reinventing less wheels.

• backups

  Keeping raw data makes them trivial

performance concerns

You can find more specific examples along with the motivation and explanations here:

• Reddit
• Instapaper/Endomondo
• Pocket
• Chrome

I run most of my data exports at least daily.

I wrote a wholepost on scheduling and job running with respect to the personal infrastructure. In short:

• on desktop: at the moment, I'm mostly using cron (to be more specific,fcron).

  I'm stillthinking of an alternative, but overall using cron is okay.

• on Android phone: I'm using Automate app and cron

This is a wrapper script I'm using to run most of my data exports.

Many things are very common to all data exports, regardless of the source. In the vast majority of cases, you want to fetch some data, save it in a file (e.g. JSON) along with a timestamp and potentially compress it.

This script aims to minimize the common boilerplate:

• path argument allows easy ISO8601 timestamping and guarantees atomic writing, so you'd never end up with corrupted exports.
• --compression allows to compress simply by passing the extension. No more tar -zcvf !
• --retries allows easy exponential backoff in case service you're querying is flaky.

Example:

arctee '/exports/rtm/{utcnow}.ical.zstd' --compression zstd --retries 3 -- /soft/export/rememberthemilk.py

1. runs /soft/export/rememberthemilk.py , retrying it up to three times if it fails

   The script is expected to dump its result in stdout; stderr is simply passed through.

2. once the data is fetched it's compressed as zstd
3. timestamp is computed and compressed data is written to /exports/rtm/20200102T170015Z.ical.zstd

The wrapper operates on regular files and is therefore, programming language agnostic as long as your export script simply outputs to stdout (or accepts a filename, so you can use /dev/stdout ). It doesn't really matter how exactly (e.g. which programming language) it's implemented.

That said, it feels kind of wrong having an extra script for all these things since they are not hard in principle, just tedious and boring to do all over again. If anyone has bright ideas on simplifying this, I'd be happy to know!