Crashing influxdb on Raspberry Pi 3+ because insufficient memory

A few days ago I’ve noticed that my influxdb installation wasn’t working properly. The server was crashing constantly.

I’ve checked the logs using

sudo journalctl -u influxdb -b

and found this

May 12 23:12:18 pi3plus influxd[30173]: ts=2019-05-12T21:12:18.440902Z lvl=info msg="Opened file" log_id=0FNU47~W000 engine=tsm1 service=filestore path=/mnt/databases/influxdb/data/_internal/monitor/342/000000020-000000002.tsm id=0 duration=14
May 12 23:12:18 pi3plus influxd[30173]: runtime: out of memory: cannot allocate 2121015296-byte block (16056320 in use)
May 12 23:12:18 pi3plus influxd[30173]: fatal error: out of memory
May 12 23:12:18 pi3plus influxd[30173]: runtime stack:
May 12 23:12:18 pi3plus influxd[30173]: runtime.throw(0xbc70be, 0xd)
May 12 23:12:18 pi3plus influxd[30173]:         /usr/local/go/src/runtime/panic.go:608 +0x5c
May 12 23:12:18 pi3plus influxd[30173]: runtime.largeAlloc(0x7e6c15dd, 0x60101, 0x76f91a20)
May 12 23:12:18 pi3plus influxd[30173]:         /usr/local/go/src/runtime/malloc.go:1021 +0x120
May 12 23:12:18 pi3plus influxd[30173]: runtime.mallocgc.func1()
May 12 23:12:18 pi3plus influxd[30173]:         /usr/local/go/src/runtime/malloc.go:914 +0x38
May 12 23:12:18 pi3plus influxd[30173]: runtime.systemstack(0x1c4e3c0)
May 12 23:12:18 pi3plus influxd[30173]:         /usr/local/go/src/runtime/asm_arm.s:354 +0x84
May 12 23:12:18 pi3plus influxd[30173]: runtime.mstart()
May 12 23:12:18 pi3plus influxd[30173]:         /usr/local/go/src/runtime/proc.go:1229
May 12 23:12:18 pi3plus influxd[30173]: goroutine 27 [running]:
May 12 23:12:18 pi3plus influxd[30173]: runtime.systemstack_switch()
May 12 23:12:18 pi3plus systemd[1]: influxdb.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 12 23:12:18 pi3plus systemd[1]: influxdb.service: Unit entered failed state.
May 12 23:12:18 pi3plus systemd[1]: influxdb.service: Failed with result 'exit-code'.

This happened because I’ve recently added statistics from my FritzBox with regards to my DSL line speed. The statistics have a high cadence, which means that many entries are created in influxdb in a short amount of time. Influxdb tries to create an index in RAM for these entries and is overwhelmed by the mass of data.

Therefore, I stopped the service with

sudo systemctl stop influxdb

and followed the suggestion from the upgrade instructions to use the influx_inspect tool.

I’ve executed influx_inspect as sudo and changed the permissions of my DB content folter later on with

chown -R influxdb:influxdb <folder>

This command may take a while to complete, depending on the size of your DB.

Once it is finished you can restart influxdb with

sudo systemctl start influxdb

Your server should now be stable again. The index is now disk based instead of being memory based, which could cause troubles on the limited resources of the Raspberry Pi.

Howto mass delete old Tweets on Twitter

There’s unfortunately no way to mass delete old Tweets you’ve posted on Twitter. There are some online services, who promise to delete your data for you, but since you’ll have to grant them access to your account I’ve got a bad feeling and wanted to do things on my own.

I’ve tried last year a windows only software called Twitter Archive Eraser. Last year it used to be a github project which you could compile locally and let it run on your account. It’s now free for a limited amount of tweets and also only works with tweets not older than two years. To remove these restrictions you’ve got to pay a small amount for a license.

You’ll need to download your complete message archive for the deletion process. Once you’ve got the data from Twitter you might as well start to write a little script which deletes the old messages for you using the Twitter post id.

Luckily, I found this blog post by Kris Shaffer. He explains how he deleted a large amount of his tweets using python so I’ve started to try this for myself. There was also a different blog which explained the process more beginner friendly. However, I’ve got problems with misformatted characters so I’ve decided to post my used code as gist to github:

To use this I’ve done the following things:

  • Requested and download my account data from Twitter
  • Create a Twitter developer account
  • Created a new app to get Api keys and Access tokens
  • Installed python3 on my mac with homebrew ‚brew install python3‘
  • Installed tweepy with pip3 ‚pip3 install tweepy‘
  • Created a virtual environment for this script
  • Copied the lines in blocks into the python3 interactive shell

Please be aware that above gist only deleted the tweets from 2017 to June 2018. Please refer for other scenarios to Kris blog post (e.g. delete only mentions in a given time frame).

Auto mount NFS shares on Raspbian

I’m using influxdb on my Raspberry Pi in combination with a NFS mount. The NFS mount is on my Synology NAS and should store the database data of influxdb. Reason for this setup is that I fear that the SD card won’t survive the many write/read cycles caused by a database writing to it.

The shared folder on my Synology is configured to be accessible by various IPs in my network:

The problem with Raspbian is that I’ve tried to auto mount the NFS share on startup, so that the influxdb service can directly write to the NFS mount. 

I’ve used these settings in my /etc/fstab to mount the volume automatically:

<DS IP>:/volume1/databases /mnt/databases nfs auto,user,rw,nolock,nosuid 0 0

This doesn’t work properly since my influxdb is often dead after a restart, but if I check the mounted volumes I see the NFS volume mounted properly.

However, there’s a tool called autofs which already helped me with a similar problem on my Mac when I moved my iTunes library to the Synology share.

Install autofs using

sudo apt-get install autofs

Open the file /etc/auto.master and add something like this

/mnt    /etc/auto.databases     -nosuid,noowners

Now create a file called /etc/auto.databases with this content

databases       -fstype=nfs,user,nolock,nosuid,rw <DS IP>:/volume1/databases

Unmount the existing NFS share. Remove/comment out the line for the nfs mount in your /etc/fstab so that it doesn’t conflict with autofs. Restart autofs with

sudo service autofs restart

Now check the content of your mount point with e.g.

ls /mnt/databases

Autofs should now automatically mount the NFS share. This might take a while, which is a good sign that the mount is loaded. You can also verify with

mount

that your NFS share is mounted to e.g. /mnt/databases. If you’ll restart now, influxdb should be happy on restart. When it tries to start, autofs will see the access to the mounted folder and will mount the NFS share before influxdb can start up properly.

Configure influxDB to store its data in a different folder

The default location of the influxDB data is /var/lib/influxdb. If you want to change the location, you’ll need to configure three folders to be in a different place. The changes should be done in the file /etc/influxdb/influxdb.conf

...
[meta]
  # Where the metadata/raft database is stored
  #dir = "/var/lib/influxdb/meta"
  dir = "/mnt/databases/influxdb/meta"
...
[data]
  # The directory where the TSM storage engine stores TSM files.
  #dir = "/var/lib/influxdb/data"
  dir = "/mnt/databases/influxdb/data"

  # The directory where the TSM storage engine stores WAL files.
  #wal-dir = "/var/lib/influxdb/wal"
  wal-dir = "/mnt/databases/influxdb/wal"

I’m using this to store the data on a NFS share which is mounted automatically. If you want to keep your existing data, move the existing content of /var/lib/influxdb to the new location.

Make sure, that the new location is owned by influxdb user and group.

Improve OpenVPN security on Synology DiskStations

I’m using OpenVPN on my Synology DiskStation with certificates instead of Preshared Keys. A few days ago I’ve wanted to login to my VPN and it wasn’t working. After checking the log file I’ve seen that there were some issues with the used configuration file for OpenVPN.

Tue Nov 20 23:04:27 2018 Cipher algorithm 'TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CB' not found
Tue Nov 20 23:04:27 2018 Exiting due to fatal error

How can this be? The configuration worked for months without problems? I’ve started to remember that I’ve started to increase the security of my OpenVPN configuration using a few parameters. The Cipher algorithm is one of them. This page describes some of the changes I’ve made (unfortunately only in German).

I’ve added the tls-cipher and tls-auth options as last parameter lines to my configuration file. The synology web UI tried to parse those parameters as cipher and auth parameter when it shows those values as part of the DSM UI.

I’ve reorderded the tls-auth and tls-cipher parameter to be above the auth and cipher parameters and the DSM UI is now able to show those values correct. This will enable you to restart the OpenVPN service from the WebUI without the need to login via SSH.

How do you get supported values for auth, cipher and tls-cipher you might wonder? Just execute

openvpn --show-tls

to get the supported tls-cipher you might line up with a : separated.

openvpn --show-digests

shows you the allowed values for auth and

openvpn --show-ciphers

will show the allowed values for cipher. However, cipher and auth can also be preselected from the DSM UI.

Don’t forget to use the same values in your OpenVPN configuration on your VPN client as well, otherwise the connection won’t work.