🚀 Quickstart in Python

Overview

The ricloud‘s reference Open Source library is implemented in Python and hosted on GitHub.

The implementation is bundled with documentation and a sample script which shows how it can be used. Configuration is limited to populating a ~/.ricloud.ini configuration file with specifying a token for authentication against the API as well as a stream endpoint value to collect the service action results.

A detailed walk-through on using the library is shown in the iCloud Photo tutorial. Source and documentation for this library can be found on GitHub.

Requirements

The ricloud library has very few requirements by design. It should run on most systems with Python 2.7. In order to use listener mode there is a further requirement of MySQL 5.7 or above. See using the sample client in production for perspectives on using the Open Source library in production.

Installation

The ricloud library can be installed with a single command:

$ pip install ricloud

To use listener mode with automatic dumping of data to MySQL, the schema configuration script must be applied to a local database. This can be used to create the required schema and tables in the database:

$ mysql -u root < sql/create_schema.sql

Configuration

The API relies on a set of security credentials, which are stored in an ricloud.ini file. This package ships with a default configuration file which enables limited access to the API for demonstration purposes.

The default credentials can be overridden by creating an override file named .ricloud.ini in the user’s HOME directory. Alternately, a RICLOUD_CONF environment variable can be set, specifying the full path and filename of the configuration file.

[mysql]
# Database config is only needed for listen mode.
host = 127.0.0.1
port = 3306
database = ricloud
username = your-ricloud-mysql-user-here
password = your-ricloud-mysql-password-here

[hosts]
api_host = https://asapi.reincubate.com
asmaster_host = https://asmaster.reincubate.com
stream_host = https://aschannel.reincubate.com

[endpoints]
account_information = /account/
register_account = /register-account/
task_status = /task-status/
result_consumed = /results-consumed/

[asmaster_endpoints]
list_services = /list-services/
list_subscriptions = /list-subscriptions/
subscribe_account = /subscribe-account/
perform_2fa_challenge = /perform-2fa-challenge/
submit_2fa_challenge = /submit-2fa-challenge/
list_devices = /list-devices/
subscribe_device = /subscribe-device/
resubscribe_account = /resubscribe-account/
unsubscribe_device = /unsubscribe-device/
unsubscribe_account = /unsubscribe-account/

[stream]
# This value provided by Reincubate.
stream_endpoint = your-aschannel-stream-name-here

[auth]
# This value provided by Reincubate.
token = your-ricloud-api-access-token-here

[output]
output_directory = output

[logging]
logs_directory = logs
time_profile = False
level = WARNING

[performance]
object_store_greenlets = 50

The default ricloud.ini can be found in this repository.

Command-line parameters

The library provides a breakdown of command-line arguments if passed the --help argument:

$ python -m ricloud --help
ricloud API Client.

Usage, interactive mode:
    ricloud <account> [--password=<password>] [--timeout=<timeout>]

Options, interactive mode:
    -h --help               Show this screen.
    --timeout=<timeout>     How long should we wait for tasks to complete, in seconds. (default: 600s or 10 minutes)
    --password=<password>   The password for this account.

Usage, manager mode:
    ricloud --list-subscriptions <serivce> [--timeout=<timeout>]
    ricloud --subscribe-account <username> <password> <service> [--timeout=<timeout>]
    ricloud --perform-2fa-challenge <account_id> <device_id> [--timeout=<timeout>]
    ricloud --submit-2fa-challenge <account_id> <code> [--timeout=<timeout>]
    ricloud --resubscribe-account <account_id> <password> [--timeout=<timeout>]
    ricloud --unsubscribe-account <account_id> [--timeout=<timeout>]
    ricloud --list-devices <account_id> [--timeout=<timeout>]
    ricloud --subscribe-device <account_id> <device_id> [--timeout=<timeout>]
    ricloud --unsubscribe-device <account_id> <device_id> [--timeout=<timeout>]

Options, manager mode:
    -h --help               Show this screen.
    --timeout=<timeout>     How long should we wait for tasks to complete, in seconds. (default: 600s or 10 minutes)

Usage, listener mode:
    ricloud --listen [--timeout=<timeout>]

Options, listener mode:
    -h --help               Show this screen.
    --timeout=<timeout>     How long should we wait for tasks to complete, in seconds. (default: 600s or 10 minutes)

Interactive mode with asapi

The library’s interactive mode provides an example of how asapi can be used to access a range of datatypes in a way that is compatible with Apple’s 2FA mechanism.

To run the sample interactive script, execute the following command:

$ python -m ricloud john.appleseed@reincubate.com --password=joshua

Note

In the example above, the API’s mock data interface is being used. This can be particularly helpful when developing and testing.

Manager mode with asmaster

The ricloud sample library ships with a command-line subscription management mode. This lets clients manually manage subscriptions where needed, and demonstrates how a production implementation could be built.

Here’s a walk-through of how a user might use manager mode. Firstly, it’s important that something is listening to aschannel when doing this. One could use the sample library in –listen mode.

Let’s see what subscriptions are registered:

{ "services": [{
    "name": "iCloud",
    "actions": [{
      "description": "",
      "parameters": [{
        "type": "string",
        "description": "",
        "optional": false,
        "name": "Device",
        "slug": "device"
      }, {
        "type": "date",
        "description": "",
        "optional": true,
        "name": "Since",
        "slug": "since"
      }],
      "name": "Fetch Data",
      "execution": "Asynchronous",
      "slug": "fetch-data",
      "permissions": {
        "data": ["sms"]
      }
    }],
    "slug": "icloud"
  }],
  "stream_endpoints": [{
    "host": "aschannel.reincubate.com",
    "protocol": "https",
    "uri": "/stream/"
  }]
}
python -m ricloud --list-subscriptions icloud
{u'accounts': [], u'success': True}

There aren’t any. Let’s add one. One could add the mock data account with this line:

$ python -m ricloud --subscribe-account john.appleseed@reincubate.com joshua icloud

Instead, let’s add a real account that is protected by 2FA:

$ python -m ricloud --subscribe-account john.appleseed@reincubate.com "xxxx" icloud
{
  "message": "This account has Multi Factor Authentication enabled, please select a device to challenge.",
  "data": {
    "trusted_devices": [
      "Challenge all 2FA devices"
    ]
  },
  "account_id": 133733,
  "success": false,
  "error": "2fa-required"
}

This means that john.appleseed@reincubate.com has been assigned an account_id of 133733, and a 2FA challenge for that account is needed:

$ python -m ricloud --perform-2fa-challenge 133733 0
{u'message': u'Challenge has been submitted.', u'success': True}

This triggers the 2FA prompt on the end-user’s devices, providing an authentication code. Let’s say that is 123456.

$ python -m ricloud --submit-2fa-challenge 133733 123456
{u'account_id': 133733, u'success': True}

So authentication has been successful, and the account is subscribed. That means one can list and subscribe to devices. Let’s see which are available:

$ python -m ricloud --list-devices 133733
{u'devices': [
  {u'ios_version': u'10.2', u'name': u'iPhone 7 Plus', u'colour': u'1', u'device_name': u'Johnny\'s iP7 iOS10 Black', u'latest-backup': u'2017-01-31 22:06:06.000000', u'model': u'D111AP', u'device_tag': u'3d0d7e5fb2ce288813306e4d4636395e047a3d28', u'serial': u'ABC123AAAAAA', u'device_id': 2},
  {u'ios_version': u'10.3.1', u'name': u'iPad Pro', u'colour': u'#e4e7e8', u'device_name': u'Johnny\'s iPad', u'latest-backup': u'2017-04-22 15:39:25.000000', u'model': u'J127AP', u'device_tag': u'b39bac0d347adfaf172527f97c3a5fa3df726a3a', u'serial': u'ABC123BBBBBB', u'device_id': 3},
  {u'ios_version': u'10.3.1', u'name': u'iPhone 7 Plus', u'colour': u'1', u'device_name': u'Johnny\'s other iPhone', u'latest-backup': u'2017-04-13 21:08:47.000000', u'model': u'D111AP', u'device_tag': u'a49bfab36504be1bf563c1d1813b05efd6076717', u'serial': u'DFVDASDDSVAS', u'device_id': 4}
], u'success': True}

Let’s also check which devices are subscribed:

$ python -m ricloud --list-subscriptions
{
  "accounts": [
    {
      "status": "ok",
      "service": "icloud",
      "account_id": 133733,
      "devices": [
        {
          "status": "ok",
          "device_id": 2
        },
        {
          "status": "ok",
          "device_id": 3
        },
        {
          "status": "ok",
          "device_id": 4
        }
      ]
    }
  ]
}

This shows that none of the devices have been subscribed, so we should subscribe to one.

$ python -m ricloud --subscribe-device 133733 2
{u'account_id': 133733, u'device_id': 2, u'success': True}

With these steps complete, data from the john.appleseed@reincubate.com and the chosen device will automatically flow down aschannel as and when it is ready.

If we try to subscribe to the same account twice, an error like this will be returned:

$ python -m ricloud --subscribe-account john.appleseed@reincubate.com josha icloud
{
  "message": "You are already subscribed to receive content for this account.",
  "account_id": 133733,
  "success": false,
  "error": "asmaster-account-already-active"
}

Listener mode with aschannel

The ricloud listener is a locally-deployed listener mechanism for the API. Essentially, it lets clients install a service in their own cloud or datacentre which automatically inserts data received from the API to a local database and filesystem.

The listener automatically dechunks data from the API, inserts feed results and metadata into a local database, and stores files on the filesystem. The listener is Open Source, and can be freely modified by users. There are many strategies which can be used with it, including having it write to shared filesystem storage or even modifying it to insert data to a client’s production datastore.

As well as writing any files it receives to disk, the listener’s database has tables for all of the data that is fed to it.

  • feed: JSON data and metadata from feed modules is stored here
  • file: File pointers and metadata is stored here
  • message: Informational messages are stored here
  • system: System messages are stored here

In order to understand and act on the data being received, it is important to regularly scan these tables for new records. It may be that the client application simply deletes rows that it has already read or imported.

Note

It is not necessarily to use the ricloud listener in order to use the API. It is simply a sample implementation of an aschannel client.

To avoid the need for a poll-based integration such as the above, clients could instead hook into the stream directly for event-based notifications, patch the listener’s code, or configure triggers in the schema.

Running

The listener process can be started with the following command:

$ python -m ricloud --listen

Important

The listener may only have a single instance running at any one time for a given key. Each time it is executed, any previously running instances will be disconnected. See simultaneous stream consumption for approaches to running the process in a high availability configuration.

Interpreting messages and system notices

Messages and system notices are automatically stored in two tables: messages and system. These messages are stored after being sent via the stream. Most of these messages are purely informational, but some are critical.

Messages

  • User token has expired; token refresh required.
  • User account locked; token refresh required.
  • Other error accessing data; no action required.

Warning

This format is subject to change.

System notices

The most common system messages are described in system messages. They will indicate any problems continuing to read from the stream.

Warning

This format is subject to change.

Accessing feed data

The listener stores each feed it receives in the feed table. The key fields are as below:

  • received: The date the feed was retrieved.
  • account_id: The account ID pertaining to the feed.
  • device_id: The device ID pertaining to the feed.
  • body: The feed in JSON format.

Warning

This format is subject to change.

Retrieving files

The listener create an entry in the database for each file it receives, and writes files to the filesystem in the output/files folder, relative to the executing shell’s current path.

mysql> SELECT id, received, account_id, device_id, location, file_id FROM file LIMIT 1000;
+----+---------------------+------------+-----------+--------------------------------------+---------+
| id | received            | account_id | device_id | location                             | file_id |
+----+---------------------+------------+-----------+--------------------------------------+---------+
|  1 | 2017-04-14 15:38:39 |       NULL |      NULL | 91f49a4f-3368-4350-ae90-a944d4652fbe |         |
|  2 | 2017-04-14 15:38:40 |       NULL |      NULL | 9b0381b6-244d-4d82-8198-57798d89a379 |         |
|  3 | 2017-04-14 15:38:41 |       NULL |      NULL | b9ccd1ed-8d80-4b33-be99-a3a7f1af73ce |         |
|  4 | 2017-04-14 15:38:42 |       NULL |      NULL | b3666148-a17e-4ca8-90db-975e1850a72b |         |
+----+---------------------+------------+-----------+--------------------------------------+---------+
4 rows in set (0.00 sec)
$ ls -l output/files
-rw-r--r-- 1 renate 1143 Apr 14 13:47 91f49a4f-3368-4350-ae90-a944d4652fbe
-rw-r--r-- 1 renate 2866 Apr 14 14:09 9b0381b6-244d-4d82-8198-57798d89a379
-rw-r--r-- 1 renate 2866 Apr 14 15:18 b9ccd1ed-8d80-4b33-be99-a3a7f1af73ce
-rw-r--r-- 1 renate 1143 Apr 14 13:49 b3666148-a17e-4ca8-90db-975e1850a72b

Files are only given context by feed data, so the place to start is with a file_id referenced by a feed entry.

Important

Each file_id is not unique! A given path will always have the same file_id. Consequently, many users and devices can lay claim to a given file_id, and they can be overwritten over time. As such, it is critical to respect the account_id and device_id values.

The file table in the database has the following key fields:

  • received: The date the file was retrieved.
  • account_id: The account ID pertaining to the file.
  • device_id: The device ID pertaining to the file.
  • location: The location of the file on the listener’s disk.
  • file_id: The ID of the file.

Warning

This format is subject to change.

Note

Whilst writing files to a single folder works well in low-volume use-cases, clients with higher volumes may wish to patch the use to use a hashed filesystem structure to avoid any filesystem limitations, or – perhaps better – to simply store the files securely in a cloud filestore.

Maintaining the listener database

None of the listener tables have any data in them that is critical to the working of the listener. Consequently, once the data has been consumed by the client application, the client can freely delete the data.

Similarly, the downloaded files may be freely deleted at any time.