Merge branch 'release-2.1'

README.md

@@ -19,20 +19,20 @@ Table of contents
     - [1.3. Tools used](#13-tools-used)
 - [2. Usage/Installation](#2-usageinstallation)
     - [2.1. Prerequisites](#21-prerequisites)
-    - [2.2. Running with Docker](#22-running-with-docker)
-    - [2.3. Environment-Variables](#23-environment-variables)
+    - [2.2. Choosing an Export-Method](#22-choosing-an-export-method)
+    - [2.3. Running with Docker](#23-running-with-docker)
+    - [2.4. Environment-Variables](#24-environment-variables)
 - [3. Data collected](#3-data-collected)
     - [3.1. Data-Types](#31-data-types)
     - [3.2. Metric-Overview](#32-metric-overview)
     - [3.3. Metric-Details](#33-metric-details)
     - [3.4. Tag-Overview](#34-tag-overview)
     - [3.5. Tag-Details](#35-tag-details)
-- [4. Screenshots](#4-screenshots)
-- [5. Potential Issues](#5-potential-issues)
-    - [5.1. Channel/Frequency](#51-channelfrequency)
-    - [5.2. Technology](#52-technology)
-    - [5.3. Data protection](#53-data-protection)
-    - [5.4. Ethical](#54-ethical)
+- [4. Potential Issues](#4-potential-issues)
+    - [4.1. Channel/Frequency](#41-channelfrequency)
+    - [4.2. Technology](#42-technology)
+    - [4.3. Data protection](#43-data-protection)
+    - [4.4. Ethical](#44-ethical)
 <!-- /TOC -->
 <br>
 
@@ -114,18 +114,34 @@ iw dev <interface> set channel <channelNumber>
 
 <br>
 
-## 2.2. Running with Docker
+## 2.2. Choosing an Export-Method
 
-### 2.2.1. Permissions
+The system allows exporting directly into [InfluxDB](https://docs.influxdata.com/influxdb) version >= 2.0 or into any system using the [InfluxDb-Line-Protocol](https://docs.influxdata.com/influxdb/v2.1/reference/syntax/line-protocol/) e.g. [QuestDB](https://questdb.io/) over TCP.
+
+As of writing (using InfluxDB v2.1 and using the *flux*-language), the data written by this system was a bit too much for InfluxDB and it struggled very quickly on a fairly beefy machine.
+
+Thats why the additional LineProtocol-Export-Method was added. Freedom of choice of the Time-Database.
+
+<br>
+
+If you want to use the InfluxDB-Line-Protocol, simply set the environment variable `USE_INFLUXDB_LINEPROTOCOL` to `true` along with the-other necessary Host and Port-variables.
+
+<br>
+
+## 2.3. Running with Docker
+
+### 2.3.1. Permissions
 
 The container must run as **root**, to have permission to listen on the wifi-interface.
 
 <br>
 
-### 2.2.2. docker run
+### 2.3.2. docker run
 
 Either run with docker directly.
 
+<details><summary>for InfluxDB</summary>
+
 ```sh
 docker run
   -d
@@ -135,18 +151,35 @@ docker run
   -e INFLUX_URL="http://influxdb:8086/"
   -e INFLUX_TOKEN="<yourToken>"
   -e INFLUX_ORG="<yourOrganisation>"
-  -e INFLUX_BUCKET="<yourBucket>"
   ruakij/rfmon-to-influx:2
 ```
+</details>
+
+<details><summary>for InfluxDB-Line-Protocol</summary>
+
+```sh
+docker run
+  -d
+  --restart unless-stopped 
+  --network host
+  -e WIFI_INTERFACE="<yourInterfaceName or leave empty for wlan0>"
+  -e USE_INFLUXDB_LINEPROTOCOL="true"
+  -e INFLUXDB_LINEPROTOCOL_HOST="<host>"
+  -e INFLUXDB_LINEPROTOCOL_PORT="<port>"
+  ruakij/rfmon-to-influx:2
+```
+</details>
 
 <br>
 
-### 2.2.3. docker-compose
+### 2.3.3. docker-compose
 
 Or use the more preferred way with docker-compose.
 
 `docker-compose.yml`
 
+<details><summary>for InfluxDB</summary>
+
 ```yaml
 version: '3'
 
@@ -163,6 +196,28 @@ services:
       - INFLUX_ORG="<yourOrganisation>"
       - INFLUX_BUCKET="<yourBucket>"
 ```
+</details>
+
+<details><summary>for InfluxDB-Line-Protocol</summary>
+
+```yaml
+version: '3'
+
+services:
+  rfmon:
+    container_name: rfmon
+    image: ruakij/rfmon-to-influx:2
+    restart: unless-stopped
+    network_mode: "host"
+    environment:
+      - WIFI_INTERFACE="<yourInterfaceName or leave empty for wlan0>"
+      - USE_INFLUXDB_LINEPROTOCOL="true"
+      - INFLUXDB_LINEPROTOCOL_HOST="<host>"
+      - INFLUXDB_LINEPROTOCOL_PORT="<port>"
+```
+</details>
+
+<br>
 
 And then pull&start the container:
 ```sh
@@ -171,9 +226,11 @@ docker-compose up -d
 
 <br>
 
-## 2.3. Environment-Variables
+## 2.4. Environment-Variables
 
-### 2.3.1. Necessary
+### 2.4.1. Necessary
+
+<details><summary>for InfluxDB</summary>
 
 Variable|Description
 ---|---
@@ -181,10 +238,20 @@ Variable|Description
 `INFLUX_TOKEN`  | Token with write-access
 `INFLUX_ORG`    | Organisation and..
 `INFLUX_BUCKET` | Bucket to write into
+</details>
+
+<details><summary>for InfluxDB-Line-Protocol</summary>
+
+Variable|Description
+---|---
+`USE_INFLUXDB_LINEPROTOCOL`     | Enable LineProtocol
+`INFLUXDB_LINEPROTOCOL_HOST`    | Host and..
+`INFLUXDB_LINEPROTOCOL_PORT`    | Port of your server
+</details>
 
 <br>
 
-### 2.3.2. Optional
+### 2.4.2. Optional
 
 Variable|Default|Description
 ---|---|---
@@ -308,13 +375,9 @@ Unknown             | -     | Unknown packets not identified into above types
 
 <br>
 
-# 4. Screenshots
+# 4. Potential Issues
 
-<br>
-
-# 5. Potential Issues
-
-## 5.1. Channel/Frequency
+## 4.1. Channel/Frequency
 
 The System can only monitor one channel at a time which might not be enough cover,
 to combat this, more Interfaces and Systems can be deployed.
@@ -323,14 +386,14 @@ This is not entirely unproblematic, as the system cannot currently prevent packa
 
 <br>
 
-## 5.2. Technology
+## 4.2. Technology
 
 Mismatches between sender and receiver-technologies (e.g. MIMO or HT) can cause packets not being logged at all.
 Though this should only be a problem for data-packets.
 
 <br>
 
-## 5.3. Data protection
+## 4.3. Data protection
 
 Because the system collects any data, this can be problematic, specially in countries with strong data-protection laws.
 
@@ -338,7 +401,7 @@ A wifi MAC address is likely to be considered as information of an identifiable
 
 <br>
 
-## 5.4. Ethical
+## 4.4. Ethical
 
 The large-scale collection of data for behavioural or movement analysis, especially without consent of the data subject, is highly controversial.
 

src/streamHandler/PacketStreamFactory.js

@@ -47,8 +47,14 @@ class PacketStreamFactory extends Transform{
     }
 
     _transform(chunk, encoding, next){
-        let packet = new Packet();
+        if(!chunk){
+            const err = "Chunk was invalid!";
+            logger.error(err);
+            next(err);
+        }
 
+        let packet = new Packet();
+        
         const lines = chunk.split("\n");
         const header = lines.splice(0, 1)[0];       // Grab first line, "lines" is now the payload
         packet = this._handleHeader(packet, header);