README.md

@@ -1,5 +1,5 @@
 # Fediverse Stats
-This code gets all peers from mastodon.social. Goal is to collect maximum number
+This code gets all peers from running Mastodon, Pleroma and Lemmy host servers and then all peers from host server's peers. Goal is to collect maximum number
 of alive fediverse's servers and then query their API to obtain their registered users (if their API provide such information).
 At the end it post the results to host server bot account.
 
@@ -7,7 +7,7 @@ At the end it post the results to host server bot account.
 
 -   **Python 3**
 -   Postgresql server
--   Mastodon running server.
+-   Mastodon or Pleroma running server.
 
 ### Usage:
 
@@ -15,21 +15,66 @@ Within Python Virtual Environment:
 
 1. Run `pip install -r requirements.txt` to install needed libraries.
 
-2. Run `python fetchservers.py` to add servers to alive servers database.
+2. Run `python db-setup.py` to setup and create new Postgresql database and needed tables in it.
 
-3. Run `python fediverse.py` to query world alive servers API. It gets data from server's nodeinfo.
+3. Run `python setup.py` to get your bot's access token of your Mastodon or Pleroma server existing account. It will be saved to 'secrets/secrets.txt' for further use.
 
-4. Use your favourite scheduling method to set `python fediverse.py` to run twice daily, `python fetchservers.py` one time daily, `python fediquery.py` to run every minute and `python uptime.py' every minute to publish best fediverse uptime.
+4. Run `python getworld.py` to get all peers from your host and the whole world of fediverse's servers (or almost the whole world).
 
-18.2.2021 - New feature! Added [Lemmy project](https://join.lemmy.ml)  
-12.5.2021 - New feature! Added Wordpress support. The code can now detect Wordpress instances with ActivityPub enabled plugin.  
-12.5.2021 - New feature! New shinny creation of servers and users graphs.  
-21.8.2021 - New feature! Added Best Fediverse's servers Uptime publishing bot.  
-22.10.2021 - New feature! Added [Funkwhale](https://funkwhale.audio) support.  
-26.10.2021 - New feature! Added [Socialhome](https://socialhome.network) support.  
-2.3.2022 - Improved server nodeinfo detection.  
-4.1.2023 - Refactored.  
-4.1.2023 - Now peers are obtained from mastodon.social's peers list.  
-5.1.2023 - Added fediquery.py. Allow ask the main bot fediverse about `soft` and `server` and it replies to the asking user with its data if any.  
-7.1.2023 - When querying a not found server, we will be added to database if it's alive.  
-8.1.2023 - Save MAU to database.
+5. Run `python fetchservers.py` to add servers to alive servers database.
+
+6. Run `python fediverse.py` to query world alive servers API. It gets data from server's API according this table:
+
+|    Software    |       API peers       |        API users (nodeinfo/2.0.json)       |                                    API users (nodeinfo/2.0)                                    | API users (api/v1/instance) | API users (main/nodeinfo/2.0) | API users (api/nodeinfo/2.0.json) |   API users (api/nodeinfo)  |    Software    |
+|:--------------:|:---------------------:|:------------------------------------------:|:----------------------------------------------------------------------------------------------:|:---------------------------:|:-----------------------------:|:---------------------------------:|:---------------------------:|:--------------:|
+| Diaspora       |                       |         ['usage']['users']['total']        |                                                                                                |                             |                               |                                   |                             | Diaspora       |
+| Friendica      | api/v1/instance/peers |                                            |                                   ['usage']['users']['total']                                  |                             |                               |                                   |                             | Friendica      |
+| Ganggo         |                       |                                            |                                   ['usage']['users']['total']                                  |                             |                               |                                   |                             | Ganggo         |
+| GNU Social     |                       |                                            |                                                                                                |                             |  ['usage']['users']['total']  |                                   |                             | GNU Social     |
+| GNU Social 2.x |                       |                                            |                                                                                                |                             |                               |    ['usage']['users']['total']    |                             | GNU Social 2.x |
+| Groundpolis    |                       |                                            |                                   ['usage']['users']['total']                                  |                             |                               |                                   |                             | Groundpolis    |
+| Hubzilla       |                       |                                            |                                   ['usage']['users']['total']                                  |                             |                               |                                   |                             | Hubzilla       |
+| Lemmy          | api/v2/site ['federated_instances']['linked']          |  ['usage']['users']['total']|                                                                    |                             |                               |                                   |                             | Lemmy          |
+| Mastodon       | api/v1/instance/peers | ['usage']['users']['total'] (since v3.0.0) |                                                                                                |   ['stats']['user_count']   |                               |                                   |                             | Mastodon       |
+| Misskey        |                       |                                            | **NO** (['usage']['users'] returns {}) so using  ['usage']['users']['activeHalfyear']  instead |                             |                               |                                   |                             | Misskey        |
+| Osada          |                       |                                            |                                   ['usage']['users']['total']                                  |                             |                               |                                   |                             | Osada          |
+| Peertube       |                       |         ['usage']['users']['total']        |                                                                                                |                             |                               |                                   |                             | Peertube       |
+| Pixelfed       |                       |                                            |                                                                                                |                             |                               |    ['usage']['users']['total']    |                             | Pixelfed       |
+| Pleroma        | api/v1/instance/peers |        ['usage']['users']['total']         |                                                                                                |                             |                               |                                   |                             | Pleroma        |
+| Plume          |                       |                                            |                                   ['usage']['users']['total']                                  |                             |                               |                                   |                             | Plume          |
+| Prismo         |                       |                                            |                                   ['usage']['users']['total']                                  |                             |                               |                                   |                             | Prismo         |
+| Ravenvale      |                       |         ['usage']['users']['total']        |                                                                                                |                             |                               |                                   |                             | Ravenvale      |
+| Squus          |                       |                                            |                                   ['usage']['users']['total']                                  |                             |                               |                                   |                             | Squus          |
+| Writefreely    |                       |                                            |                                                                                                |                             |                               |                                   | ['usage']['users']['total'] | Writefreely    |
+| Zap            |                       |                                            |                                   ['usage']['users']['total']                                  |                             |                               |                                   |                             | Zap            |
+
+| software    |          API          |     software name    |
+|-------------|:---------------------:|:--------------------:|
+| diaspora    | nodeinfo/2.0.json     | ['software']['name'] |
+| dolphin     | nodeinfo/2.0          | ['software']['name'] |
+| ganggo      | nodeinfo/2.0          | ['software']['name'] |
+| friendica   | nodeinfo/2.0          | ['software']['name'] |
+| gnusocial   | main/nodeinfo/2.0     | ['software']['name'] |
+| gnusocialv2 | api/nodeinfo/2.0.json | ['software']['name'] |
+| groundpolis | nodeinfo/2.0          | ['software']['name'] |
+| hubzilla    | nodeinfo/2.0          | ['software']['name'] |
+| lemmy       | nodeinfo/2.0.json     | ['software']['name'] |
+| mastodon    | nodeinfo/2.0.json     | ['software']['name'] |
+| misskey     | nodeinfo/2.0          | ['software']['name'] |
+| osada       | nodeinfo/2.0.json     | ['software']['name'] |
+| peertube    | nodeinfo/2.0.json     | ['software']['name'] |
+| pixelfed    | api/nodeinfo/2.0.json | ['software']['name'] |
+| pleroma     | nodeinfo/2.0.json     | ['software']['name'] |
+| plume       | nodeinfo/2.0          | ['software']['name'] |
+| prismo      | nodeinfo/2.0.json     | ['software']['name'] |
+| ravenvale   | nodeinfo/2.0.json     | ['software']['name'] |
+| squs        | nodeinfo/2.0          | ['software']['name'] |
+| wordpress   | wp-json/nodeinfo/2.0  | ['software']['name'] |
+| writefreely | api/nodeinfo          | ['software']['name'] |
+| zap         | nodeinfo/2.0.json     | ['software']['name'] |
+
+5. Use your favourite scheduling method to set `python fediverse.py` to run twice daily, `python fetchservers.py` one time daily and `python getworld.py` to run monthly.   
+
+18.2.21 - New feature! Added [Lemmy project](https://join.lemmy.ml)  
+12.5.21 - New feature! Added Wordpress support. The code can now detect Wordpress instances with ActivityPub enabled plugin.  
+12.5.21 - New feature! New shinny creation of servers and users graphs.  

db-setup.py

@@ -0,0 +1,160 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+import getpass
+import os
+import sys
+from mastodon import Mastodon
+from mastodon.Mastodon import MastodonMalformedEventError, MastodonNetworkError, MastodonReadTimeout, MastodonAPIError
+import psycopg2
+from psycopg2 import sql
+from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
+
+# Returns the parameter from the specified file
+def get_parameter( parameter, file_path ):
+    # Check if secrets file exists
+    if not os.path.isfile(file_path):
+        print("File %s not found, asking."%file_path)
+        write_parameter( parameter, file_path )
+        #sys.exit(0)
+
+    # Find parameter in file
+    with open( file_path ) as f:
+        for line in f:
+            if line.startswith( parameter ):
+                return line.replace(parameter + ":", "").strip()
+
+    # Cannot find parameter, exit
+    print(file_path + "  Missing parameter %s "%parameter)
+    sys.exit(0)
+
+def write_parameter( parameter, file_path ):
+  if not os.path.exists('config'):
+    os.makedirs('config')
+  print("Setting up fediverse parameters...")
+  print("\n")
+  fediverse_db = input("fediverse db name: ")
+  fediverse_db_user = input("fediverse db user: ")
+
+  with open(file_path, "w") as text_file:
+    print("fediverse_db: {}".format(fediverse_db), file=text_file)
+    print("fediverse_db_user: {}".format(fediverse_db_user), file=text_file)
+
+def create_table(db, db_user, table, sql):
+
+  conn = None
+  try:
+
+    conn = psycopg2.connect(database = db, user = db_user, password = "", host = "/var/run/postgresql", port = "5432")
+    cur = conn.cursor()
+
+    print("Creating table.. "+table)
+    # Create the table in PostgreSQL database
+    cur.execute(sql)
+
+    conn.commit()
+    print("Table "+table+" created!")
+    print("\n")
+
+  except (Exception, psycopg2.DatabaseError) as error:
+
+    print(error)
+
+  finally:
+
+    if conn is not None:
+
+      conn.close()
+
+#############################################################################################
+
+# Load configuration from config file
+config_filepath = "config/db_config.txt"
+fediverse_db = get_parameter("fediverse_db", config_filepath)
+fediverse_db_user = get_parameter("fediverse_db_user", config_filepath)
+
+############################################################
+# create database
+############################################################
+
+conn = None
+
+try:
+
+  conn = psycopg2.connect(dbname='postgres',
+      user=fediverse_db_user, host='',
+      password='')
+
+  conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
+
+  cur = conn.cursor()
+
+  print("Creating database " + fediverse_db + ". Please wait...")
+
+  cur.execute(sql.SQL("CREATE DATABASE {}").format(
+          sql.Identifier(fediverse_db))
+      )
+  print("Database " + fediverse_db + " created!")
+
+except (Exception, psycopg2.DatabaseError) as error:
+
+  print(error)
+
+finally:
+
+  if conn is not None:
+
+    conn.close()
+
+#############################################################################################
+
+try:
+
+  conn = None
+  conn = psycopg2.connect(database = fediverse_db, user = fediverse_db_user, password = "", host = "/var/run/postgresql", port = "5432")
+
+except (Exception, psycopg2.DatabaseError) as error:
+
+  print(error)
+  # Load configuration from config file
+  os.remove("db_config.txt")
+  print("Exiting. Run db-setup again with right parameters")
+  sys.exit(0)
+
+if conn is not None:
+
+  print("\n")
+  print("fediverse parameters saved to db-config.txt!")
+  print("\n")
+
+############################################################
+# Create needed tables
+############################################################
+
+print("Creating table...")
+
+########################################
+
+db = fediverse_db
+db_user = fediverse_db_user
+table = "world"
+sql = "create table "+table+" (server varchar(200) PRIMARY KEY, federated_with varchar(200), updated_at timestamptz, saved_at timestamptz), checked boolean"
+create_table(db, db_user, table, sql)
+
+table = "fediverse"
+sql = "create table "+table+" (server varchar(200) PRIMARY KEY, users INT, updated_at timestamptz, software varchar(50), version varchar(100))"
+create_table(db, db_user, table, sql)
+
+table = "totals"
+sql = "create table "+table+" (datetime timestamptz PRIMARY KEY, total_servers INT, total_users INT)"
+create_table(db, db_user, table, sql)
+
+table = "evo"
+sql = "create table "+table+" (datetime timestamptz PRIMARY KEY, servers INT, users INT)"
+create_table(db, db_user, table, sql)
+
+#####################################
+
+print("Done!")
+print("Now you can run setup.py!")
+print("\n")

fediquery.py

@@ -1,196 +0,0 @@
-import sys
-import os
-import os.path
-import re
-from datetime import datetime, timedelta
-from setup import Setup
-from mastodon import Mastodon
-from database import Database
-from query import Query
-import pdb
-
-def cleanhtml(raw_html):
-    cleanr = re.compile('<.*?>')
-    cleantext = re.sub(cleanr, '', raw_html)
-    return cleantext
-
-def unescape(s):
-    s = s.replace("&apos;", "'")
-    return s
-
-def replying():
-
-    reply = False
-
-    content = cleanhtml(text)
-    content = unescape(content)
-
-    try:
-
-        start = content.index("@")
-        end = content.index(" ")
-        if len(content) > end:
-
-            content = content[0: start:] + content[end +1::]
-
-        neteja = content.count('@')
-
-        i = 0
-        while i < neteja :
-
-            start = content.rfind("@")
-            end = len(content)
-            content = content[0: start:] + content[end +1::]
-            i += 1
-
-        question = content.lower()
-
-        query_word = question
-        query_word_length = len(query_word)
-
-        if query_word[:4]  == 'soft':
-
-            reply = True
-
-        if query_word[:6]  == 'server':
-
-            reply = True
-
-        if query_word[:3]  == 'mau':
-
-            reply = True
-
-        return (reply, query_word)
-
-    except ValueError as v_error:
-
-        print(v_error)
-
-        query_word = ''
-
-        return (reply, query_word)
-
-# main
-
-if __name__ == '__main__':
-
-    setup = Setup()
-
-    mastodon = Mastodon(
-            access_token = setup.mastodon_app_token,
-            api_base_url= setup.mastodon_hostname
-            )
-
-    db = Database()
-
-    query = Query()
-
-    now = datetime.now()
-
-    bot_id = mastodon.me().id
-
-    notifications = mastodon.notifications()
-
-    if len(notifications) == 0:
-
-        print('No mentions')
-        sys.exit(0)
-
-    for notif in notifications:
-
-        notification_id = notif.id
-
-        if notif.type != 'mention':
-
-            print(f'dismissing notification {notification_id}')
-
-            mastodon.notifications_dismiss(notification_id)
-
-            continue
-
-        account_id = notif.account.id
-
-        username = notif.account.acct
-
-        status_id = notif.status.id
-
-        text  = notif.status.content
-
-        visibility = notif.status.visibility
-
-        reply, query_word = replying()
-
-        if reply == True:
-
-            if query_word[:4] == 'soft':
-
-                key_word = query_word[:4]
-
-                search_soft = query_word[5:]
-
-                if search_soft != '':
-
-                    servers, users, mau = db.get_soft_data(search_soft)
-
-                    toot_text = f'@{username}, my data for {search_soft} software:\n\n'
-
-                    if servers != 0:
-
-                        toot_text += f'software :{search_soft}:\nservers: {servers:,}\nusers: {users:,}\nMAU: {mau:,}'
-
-                    else:
-
-                        toot_text += 'software not found!'
-
-                    mastodon.status_post(toot_text, in_reply_to_id=status_id,visibility=visibility)
-
-                    print(f'Notification {notification_id} replied')
-
-                    mastodon.notifications_dismiss(notification_id)
-
-            if query_word[:6] == 'server':
-
-                key_word = query_word[:6]
-
-                search_server = query_word[7:]
-
-                if search_server != '':
-
-                    server, software, version, users, mau, alive = db.fediquery_server_data(search_server)
-
-                    toot_text = f'@{username}, my data for {search_server}:\n\n'
-
-                    if server == '' or server != '' and not alive:
-
-                        server, software, version, users, mau, alive = query.getsoft(search_server)
-
-                        if server != '' and alive:
-
-                            toot_text += f"\nServer not found but it's alive. Added!\n\n"
-
-                    if alive:
-
-                        toot_text += f'server: {server}\nsoftware: :{software}:\nversion: {version}\nMAU: {int(mau):,}\nusers: {int(users):,}\nalive: {alive}'
-
-                    else:
-
-                        toot_text += 'server not found!'
-
-                    mastodon.status_post(toot_text, in_reply_to_id=status_id,visibility=visibility)
-
-                    print(f'Notification {notification_id} replied')
-
-                    mastodon.notifications_dismiss(notification_id)
-        else:
-
-            try:
-
-                print(f'Dismissing notification {notification_id}')
-
-                mastodon.notifications_dismiss(notification_id)
-
-            except MastodonNotFoundError as notfound_error:
-
-                print(f'{notfound_error}')
-
-                continue

fetchservers.py

@@ -4,47 +4,121 @@ import os
 import json
 import sys
 import os.path
-from setup import Setup
-from database import Database
-import requests
-import urllib3
+import psycopg2
+from multiprocessing import Pool, Manager
+import aiohttp
+import asyncio
 import socket
-import ray
-import pdb
 
-ray.init(num_cpus = 25) # Specify this system CPUs.
+apis = ['/nodeinfo/2.0?', '/nodeinfo/2.0.json?', '/main/nodeinfo/2.0?', '/api/statusnet/config?',
+        '/api/nodeinfo/2.0.json?', '/api/nodeinfo?', '/api/v1/instance?', '/wp-json/nodeinfo/2.0?']
 
-from ray.exceptions import (
-    RaySystemError,
-    RayError,
-    RayTaskError,
-    ObjectStoreFullError,
+client_exceptions = (
+    aiohttp.ClientResponseError,
+    aiohttp.ClientConnectionError,
+    aiohttp.ClientConnectorError,
+    aiohttp.ClientError,
+    asyncio.TimeoutError,
+    socket.gaierror,
 )
 
-apis = ['/api/v1/instance',
-        '/api/v1/nodeinfo',
-        '/nodeinfo/2.0',
-        '/nodeinfo/2.0.json',
-        '/nodeinfo/2.1.json',
-        '/main/nodeinfo/2.0',
-        '/api/statusnet/config',
-        '/api/nodeinfo/2.0.json',
-        '/api/nodeinfo',
-        '/wp-json/nodeinfo/2.0',
-        '/api/v1/instance/nodeinfo/2.0',
-        '/.well-known/x-nodeinfo2'
-        ]
 
 def is_json(myjson):
-
     try:
         json_object = json.loads(myjson)
     except ValueError as e:
         return False
     return True
 
-@ray.remote
-def getsoft(server):
+
+def write_api(server, software, users, alive, api, soft_version):
+    insert_sql = "INSERT INTO fediverse(server, updated_at, software, users, alive, users_api, version) VALUES(%s,%s,%s,%s,%s,%s,%s) ON CONFLICT DO NOTHING"
+    conn = None
+
+    try:
+
+        conn = psycopg2.connect(database=fediverse_db, user=fediverse_db_user, password="", host="/var/run/postgresql",
+                                port="5432")
+
+        cur = conn.cursor()
+
+        cur.execute(insert_sql, (server, now, software, users, alive, api, soft_version))
+
+        cur.execute(
+            "UPDATE fediverse SET updated_at=(%s), software=(%s), users=(%s), alive=(%s), users_api=(%s), version=(%s) where server=(%s)",
+            (now, software, users, alive, api, soft_version, server))
+
+        cur.execute("UPDATE world SET checked='t' where server=(%s)", (server,))
+
+        conn.commit()
+
+        cur.close()
+
+    except (Exception, psycopg2.DatabaseError) as error:
+
+        print(error)
+
+    finally:
+
+        if conn is not None:
+            conn.close()
+
+
+async def getsoft(server):
+    try:
+
+        socket.gethostbyname(server)
+
+    except socket.gaierror:
+
+        pass
+        return
+
+    soft = ''
+
+    url = 'https://' + server
+
+    timeout = aiohttp.ClientTimeout(total=3)
+    async with aiohttp.ClientSession(timeout=timeout) as session:
+        for api in apis:
+            try:
+                async with session.get(url + api) as response:
+                    if response.status == 200:
+                        try:
+                            response_json = await response.json()
+                        except:
+                            pass
+            except aiohttp.ClientConnectorError as err:
+                pass
+            else:
+                if response.status == 200 and api != '/api/v1/instance?':
+                    try:
+                        soft = response_json['software']['name']
+                        soft = soft.lower()
+                        soft_version = response_json['software']['version']
+                        users = response_json['usage']['users']['total']
+                        if users > 1000000:
+                            return
+                        alive = True
+                        write_api(server, soft, users, alive, api, soft_version)
+                        print("Server " + server + " (" + soft + " " + soft_version + ") is alive!")
+                        return
+                    except:
+                        pass
+                if response.status == 200 and soft == '' and api == "/api/v1/instance?":
+                    soft = 'mastodon'
+                    users = response_json['stats']['user_count']
+                    soft_version = response_json['version']
+                    if users > 1000000:
+                        return
+                    alive = True
+                    write_api(server, soft, users, alive, api)
+                    print("Server " + server + " (" + soft + ") is alive!")
+
+
+def getserver(server, x):
+
+    server = server[0].rstrip('.').lower()
 
     if server.find(".") == -1:
         return
@@ -55,281 +129,94 @@ def getsoft(server):
     if server.find(":") != -1:
         return
 
-    soft = ''
-
-    is_nodeinfo = False
-
-    url = 'https://' + server
-
     try:
 
-        response = requests.get(url + '/.well-known/nodeinfo', headers = setup.user_agent, timeout=3)
+        loop = asyncio.get_event_loop()
+        coroutines = [getsoft(server)]
+        soft = loop.run_until_complete(asyncio.gather(*coroutines, return_exceptions=True))
 
-        if response.status_code == 200:
-
-            try:
-
-                response_json = response.json()
-
-                if len(response_json['links']) == 1:
-
-                    nodeinfo = response_json['links'][0]['href'].replace(f'https://{server}','')
-
-                elif len(response_json['links']) == 2:
-
-                    nodeinfo = response_json['links'][1]['href'].replace(f'https://{server}','')
-
-                try:
-
-                    nodeinfo_data = requests.get(url + nodeinfo, headers = setup.user_agent, timeout=3)
-
-                    if nodeinfo_data.status_code == 200:
-
-                        nodeinfo_json = nodeinfo_data.json()
-
-                        is_nodeinfo = True
-
-                    else:
-
-                        print(f"{nodeinfo} not responding: error code {nodeinfo_data.status_code}")
-
-                except:
-
-                    pass
-
-            except:
-
-                print(f'{server} is not responding: error code {response.status_code}')
-                print('*********************************************************************')
-
-                pass
-        else:
-
-            for api in apis:
-
-                try:
-
-                    response = requests.get(url + api, headers = setup.user_agent, timeout=3)
-
-                    if response.status_code == 200:
-
-                        if is_json(response.text):
-
-                            nodeinfo_json = response.json()
-
-                            if 'software' in nodeinfo_json:
-
-                                nodeinfo = api
-
-                                is_nodeinfo = True
-
-                                break
-
-                            elif 'title' in nodeinfo_json:
-
-                                if nodeinfo_json['title'] == 'Zap':
-
-                                    nodeinfo = api
-
-                                    is_nodeinfo = True
-
-                                    soft = 'zap'
-
-                                    break
-
-                            elif 'version' in nodeinfo_json:
-
-                                nodeinfo = api
-
-                                is_nodeinfo = True
-
-                                break
-
-                except:
-
-                    pass
-
-    except requests.exceptions.SSLError as errssl:
+    except:
 
         pass
 
-    except requests.exceptions.HTTPError as errh:
 
-        pass
+# Returns the parameter from the specified file
+def get_parameter(parameter, file_path):
+    # Check if secrets file exists
+    if not os.path.isfile(file_path):
+        print("File %s not found, exiting." % file_path)
+        sys.exit(0)
 
-    except requests.exceptions.ConnectionError as errc:
+    # Find parameter in file
+    with open(file_path) as f:
+        for line in f:
+            if line.startswith(parameter):
+                return line.replace(parameter + ":", "").strip()
 
-        pass
+    # Cannot find parameter, exit
+    print(file_path + "  Missing parameter %s " % parameter)
+    sys.exit(0)
 
-    except requests.exceptions.ReadTimeout as to_err:
 
-        pass
+# Load configuration from config file
+config_filepath = "config/config.txt"
+mastodon_hostname = get_parameter("mastodon_hostname", config_filepath)
 
-    except requests.exceptions.TooManyRedirects as tmr_err:
-
-        pass
-
-    except urllib3.exceptions.LocationParseError as lp_err:
-
-        pass
-
-    except requests.exceptions.InvalidURL as iu_err:
-
-        pass
-
-    except requests.exceptions.ChunkedEncodingError as chunk_err:
-
-        print(f'ChunkedEncodingError! {server}')
-        pass
-
-    except ray.exceptions.RaySystemError as ray_sys_error:
-
-        print(ray_sys_error) 
-        pass
-
-    else:
-
-        if is_nodeinfo:
-
-            if nodeinfo != '/api/v1/instance?':
-
-                if nodeinfo != '/.well-known/x-nodeinfo2?':
-
-                    try:
-
-                        soft = nodeinfo_json['software']['name']
-                        soft = soft.lower()
-                        soft_version = nodeinfo_json['software']['version']
-                        users = nodeinfo_json.get('usage').get('users').get('total') or '0'
-                        if int(users) > 1000000:
-                            return
-                        self.mau = nodeinfo_json.get('usage').get('users').get('activeMonth') or 0
-                        alive = True
-
-                        db.write_api(server, soft, users, mau, alive, nodeinfo, soft_version)
-
-                        print(f"Server {server} ({soft} {soft_version}) is alive!")
-                        print('*********************************************************************')
-
-                        return
-
-                    except:
-
-                        pass
-
-                else:
-
-                    try:
-
-                        soft = nodeinfo_json['server']['software']
-                        soft = soft.lower()
-                        soft_version = nodeinfo_json['server']['version']
-                        users = nodeinfo_json['usage']['users']['total']
-                        if int(users) > 1000000:
-                            return
-                        self.mau = nodeinfo_json.get('usage').get('users').get('activeMonth') or 0
-                        alive = True
-
-                        if soft == 'socialhome':
-
-                            db.write_api(server, soft, users, mau, alive, nodeinfo, soft_version)
-
-                            print('*********************************************************************')
-                            print(f"Server {serve}r ({soft} {soft_version}) is alive!")
-                            print('*********************************************************************')
-
-                        return
-
-                    except:
-
-                        pass
-
-            if soft == '' and nodeinfo == "/api/v1/instance?":
-
-                soft = 'mastodon'
-
-                try:
-
-                    users = nodeinfo_json['stats']['user_count']
-
-                    if int(users) > 1000000:
-
-                        return
-
-                except:
-
-                    users = 0
-
-                try:
-
-                    soft_version = nodeinfo_json['version']
-
-                except:
-
-                    soft_version = 'unknown'
-
-                mau = 0
-                alive = True
-
-                db.write_api(server, soft, users, mau, alive, nodeinfo, soft_version)
-
-                print('*********************************************************************')
-                print(f"Server {server} ({soft}) is alive!")
-
-            elif soft == 'zap' and nodeinfo == "/api/v1/instance?":
-
-                soft = 'zap'
-                users = nodeinfo_json['stats']['user_count']
-                soft_version = nodeinfo_json['version']
-                alive = True
-
-                print(server, soft, users, alive, api)
-
-                print('*********************************************************************')
-                print(f"Server {server} ({soft}) is alive!")
-
-        else:
-
-            print(f'Server {server} is dead')
-            print('*********************************************************************')
+# Load database config from db_config file
+db_config_filepath = "config/db_config.txt"
+fediverse_db = get_parameter("fediverse_db", db_config_filepath)
+fediverse_db_user = get_parameter("fediverse_db_user", db_config_filepath)
 
 ###############################################################################
 # main
 
 if __name__ == '__main__':
 
-    ## name: fetchservers.py
+    now = datetime.now()
+    start_time = time.time()
 
-    setup = Setup()
-
-    db = Database()
-
-    res = requests.get('https://' + 'mastodon.social' + setup.peers_api, headers = setup.user_agent, timeout=3)
-
-    hostname_peers = res.json()
-
-    start = datetime.now()
-
-    program = 'fetchservers'
-
-    finish = start
-
-    db.save_time(program, start, finish)
-
-    now = start
-
-    ray_start = time.time()
+    world_servers = []
 
     try:
 
-        results = ray.get([getsoft.remote(server) for server in hostname_peers])
+        conn = None
 
-        print(f"duration = {time.time() - ray_start}.\nprocessed servers: {len(results)}")
+        conn = psycopg2.connect(database=fediverse_db, user=fediverse_db_user, password="", host="/var/run/postgresql",
+                                port="5432")
 
-    except:
+        cur = conn.cursor()
 
-        pass
+        # get world servers list
 
-    finish = datetime.now()
+        cur.execute("select server from world where checked='f'")
 
-    db.save_time(program, start, finish)
+        for row in cur:
+            world_servers.append(row[0])
+
+        cur.close()
+
+        print("Remaining servers: " + str(len(world_servers)))
+
+    except (Exception, psycopg2.DatabaseError) as error:
+
+        print(error)
+
+    finally:
+
+        if conn is not None:
+            conn.close()
+
+    ###########################################################################
+    # multiprocessing!
+
+    m = Manager()
+    q = m.Queue()
+    z = zip(world_servers)
+
+    serv_number = len(world_servers)
+
+    pool_tuple = [(x, q) for x in z]
+    with Pool(processes=64) as pool:
+        pool.starmap(getserver, pool_tuple)
+
+    print('Done.')

getworld.py

@@ -0,0 +1,282 @@
+import time
+start_time = time.time()
+from six.moves import urllib
+from datetime import datetime
+from subprocess import call
+from mastodon import Mastodon
+import threading
+import os
+import json
+import signal
+import sys
+import os.path
+import requests
+import operator
+import calendar
+import psycopg2
+from itertools import product
+
+from multiprocessing import Pool, Lock, Process, Queue, current_process
+import queue
+import multiprocessing
+
+import aiohttp
+import aiodns
+import asyncio
+from aiohttp import ClientError, ClientSession, ClientConnectionError, ClientConnectorError, ClientSSLError, ClientConnectorSSLError, ServerTimeoutError
+from asyncio import TimeoutError
+import socket
+from socket import gaierror, gethostbyname
+
+updated_at = datetime.now()
+peers_api = '/api/v1/instance/peers?'
+lemmy_api = '/api/v2/site?'
+
+def is_json(myjson):
+  try:
+    json_object = json.loads(myjson)
+  except ValueError as e:
+    return False
+  return True
+
+def get_lemmy_server(server):
+
+    if server.find(".") == -1:
+        return
+    if server.find("@") != -1:
+       return
+    if server.find("/") != -1:
+       return
+    if server.find(":") != -1:
+       return
+
+    try:
+
+        loop = asyncio.get_event_loop()
+        coroutines = [get_lemmy_peers(server)]
+        loop.run_until_complete(asyncio.gather(*coroutines, return_exceptions=True))
+
+    except:
+
+        pass
+
+def getserver(server):
+
+    if server.find(".") == -1:
+        return
+    if server.find("@") != -1:
+       return
+    if server.find("/") != -1:
+       return
+    if server.find(":") != -1:
+       return
+
+    try:
+
+        loop = asyncio.get_event_loop()
+        coroutines = [getpeers(server)]
+        loop.run_until_complete(asyncio.gather(*coroutines, return_exceptions=True))
+
+    except:
+
+        pass
+
+async def get_lemmy_peers(server):
+
+    try:
+
+        socket.gethostbyname(server)
+
+    except socket.gaierror:
+
+        return
+
+    url = 'https://' + server
+
+    timeout = aiohttp.ClientTimeout(total=3)
+    async with aiohttp.ClientSession(timeout=timeout) as session:
+
+            try:
+                async with session.get(url+lemmy_api) as resp:
+                    response = await resp.json()
+                    if resp.status == 200:
+                        try:
+                            data = response['federated_instances']['linked']
+                            print("Server: " + server + ", " + "federated with " + str(len(data)) + " servers")
+                            i = 0
+                            while i < len(data):
+
+                                saved_at = datetime.now()
+                                insert_sql = "INSERT INTO world(server, federated_with, updated_at, saved_at) VALUES(%s,%s,%s,%s) ON CONFLICT DO NOTHING"
+                                conn = None
+                                try:
+
+                                    conn = psycopg2.connect(database = fediverse_db, user = fediverse_db_user, password = "", host = "/var/run/postgresql", port = "5432")
+
+                                    cur = conn.cursor()
+
+                                    cur.execute(insert_sql, (data[i], server, updated_at, saved_at,))
+
+                                    conn.commit()
+
+                                    cur.close()
+
+                                except (Exception, psycopg2.DatabaseError) as error:
+
+                                    print(error)
+
+                                finally:
+
+                                    if conn is not None:
+
+                                        conn.close()
+
+                                i += 1
+
+                        except:
+
+                            pass
+
+            except aiohttp.ClientConnectorError as err:
+
+                pass
+
+
+async def getpeers(server):
+
+    try:
+
+        socket.gethostbyname(server)
+
+    except socket.gaierror:
+
+        return
+
+    url = 'https://' + server
+
+    timeout = aiohttp.ClientTimeout(total=3)
+
+    async with aiohttp.ClientSession(timeout=timeout) as session:
+
+            try:
+
+                async with session.get(url+peers_api) as resp:
+
+                    response = await resp.json()
+
+                    if resp.status == 200:
+
+                        try:
+
+                            response_json = response
+
+                            print("Server: " + server + ", " + "federated with " + str(len(response_json)) + " servers")
+
+                            i = 0
+
+                            while i < len(response_json):
+
+                                saved_at = datetime.now()
+
+                                insert_sql = "INSERT INTO world(server, federated_with, updated_at, saved_at) VALUES(%s,%s,%s,%s) ON CONFLICT DO NOTHING"
+
+                                conn = None
+
+                                try:
+
+                                    conn = psycopg2.connect(database = fediverse_db, user = fediverse_db_user, password = "", host = "/var/run/postgresql", port = "5432")
+
+                                    cur = conn.cursor()
+
+                                    cur.execute(insert_sql, (response_json[i], server, updated_at, saved_at,))
+
+                                    conn.commit()
+
+                                    cur.close()
+
+                                except (Exception, psycopg2.DatabaseError) as error:
+
+                                    print(error)
+
+                                finally:
+
+                                    if conn is not None:
+
+                                        conn.close()
+
+                                i += 1
+
+                        except:
+
+                            pass
+
+            except aiohttp.ClientConnectorError as err:
+
+                pass
+
+###############################################################################
+# INITIALISATION
+###############################################################################
+
+# Returns the parameter from the specified file
+def get_parameter( parameter, file_path ):
+    # Check if secrets file exists
+    if not os.path.isfile(file_path):
+        print("File %s not found, exiting."%file_path)
+        sys.exit(0)
+
+    # Find parameter in file
+    with open( file_path ) as f:
+        for line in f:
+            if line.startswith( parameter ):
+                return line.replace(parameter + ":", "").strip()
+
+    # Cannot find parameter, exit
+    print(file_path + "  Missing parameter %s "%parameter)
+    sys.exit(0)
+
+# Load secrets from secrets file
+secrets_filepath = "secrets/secrets.txt"
+uc_client_id     = get_parameter("uc_client_id",     secrets_filepath)
+uc_client_secret = get_parameter("uc_client_secret", secrets_filepath)
+uc_access_token  = get_parameter("uc_access_token",  secrets_filepath)
+
+# Load configuration from config file
+config_filepath = "config/config.txt"
+mastodon_hostname = get_parameter("mastodon_hostname", config_filepath)
+
+# Load database config from db_config file
+db_config_filepath = "config/db_config.txt"
+fediverse_db = get_parameter("fediverse_db", db_config_filepath)
+fediverse_db_user = get_parameter("fediverse_db_user", db_config_filepath)
+
+# Initialise Mastodon API
+mastodon = Mastodon(
+    client_id = uc_client_id,
+    client_secret = uc_client_secret,
+    access_token = uc_access_token,
+    api_base_url = 'https://' + mastodon_hostname,
+)
+
+# Initialise access headers
+headers={ 'Authorization': 'Bearer %s'%uc_access_token }
+
+###############################################################################
+# main
+
+if __name__ == '__main__':
+
+    lemmy_server = 'lemmy.ml'
+    get_lemmy_server(lemmy_server)
+
+    getserver(mastodon_hostname)
+    self_peers = mastodon.instance_peers()
+
+    ###########################################################################
+
+    nprocs = multiprocessing.cpu_count()
+    with multiprocessing.Pool(processes=nprocs) as pool:
+        results = pool.starmap(getserver, product(self_peers))
+
+    exec_time = str(round((time.time() - start_time), 2))
+    print(exec_time)

query.py

@@ -1,300 +0,0 @@
-import time
-from datetime import datetime
-import os
-import json
-import sys
-import os.path
-import requests
-import urllib3
-import socket
-from database import Database
-from setup import Setup
-import pdb
-
-apis = ['/api/v1/instance?',
-        '/api/v1/nodeinfo?',
-        '/nodeinfo/2.0?',
-        '/nodeinfo/2.0.json?',
-        '/nodeinfo/2.1.json?',
-        '/main/nodeinfo/2.0?',
-        '/api/statusnet/config?',
-        '/api/nodeinfo/2.0.json?',
-        '/api/nodeinfo?',
-        '/wp-json/nodeinfo/2.0?',
-        '/api/v1/instance/nodeinfo/2.0?',
-        '/.well-known/x-nodeinfo2?'
-        ]
-
-def is_json(myjson):
-
-    try:
-        json_object = json.loads(myjson)
-    except ValueError as e:
-        return False
-    return True
-
-class Query():
-
-    name = "Query server data"
-
-    def __init__(self, server=None, db=None, setup=None, soft=None, soft_version=None, version=None, users=None, mau=None, alive=False):
-
-        self.server = server
-        self.db = Database()
-        self.setup = Setup()
-        self.soft = ''
-        self.soft_version = ''
-        self.users = 0
-        self.mau = 0
-        self.alive = alive
-
-    def getsoft(self, server):
-
-        if server.find(".") == -1:
-            return
-        if server.find("@") != -1:
-            return
-        if server.find("/") != -1:
-            return
-        if server.find(":") != -1:
-            return
-
-        is_nodeinfo = False
-
-        url = 'https://' + server
-
-        try:
-
-            response = requests.get(url + '/.well-known/nodeinfo', headers = self.setup.user_agent, timeout=3)
-
-            if response.status_code == 200:
-
-                try:
-
-                    response_json = response.json()
-
-                    if len(response_json['links']) == 1:
-
-                        nodeinfo = response_json['links'][0]['href'].replace(f'https://{server}','')
-
-                    elif len(response_json['links']) == 2:
-
-                        nodeinfo = response_json['links'][1]['href'].replace(f'https://{server}','')
-
-                    try:
-
-                        nodeinfo_data = requests.get(url + nodeinfo, headers = self.setup.user_agent, timeout=3)
-
-                        if nodeinfo_data.status_code == 200:
-
-                            nodeinfo_json = nodeinfo_data.json()
-
-                            is_nodeinfo = True
-
-                        else:
-
-                            print(f"{nodeinfo} not responding: error code {nodeinfo_data.status_code}")
-
-                    except:
-
-                        pass
-
-                except:
-
-                    print(f'{server} is not responding: error code {response.status_code}')
-                    print('*********************************************************************')
-
-                    pass
-            else:
-
-                for api in apis:
-
-                    try:
-
-                        response = requests.get(url + api, headers = self.setup.user_agent, timeout=3)
-
-                        if response.status_code == 200:
-
-                            if is_json(response.text):
-
-                                nodeinfo_json = response.json()
-
-                                if 'software' in nodeinfo_json:
-
-                                    nodeinfo = api
-
-                                    is_nodeinfo = True
-
-                                    break
-
-                                elif 'title' in nodeinfo_json:
-
-                                    if nodeinfo_json['title'] == 'Zap':
-
-                                        nodeinfo = api
-
-                                        is_nodeinfo = True
-
-                                        soft = 'zap'
-
-                                        break
-
-                                elif 'version' in nodeinfo_json:
-
-                                    nodeinfo = api
-
-                                    is_nodeinfo = True
-
-                                    break
-
-                    except:
-
-                        pass
-
-        except requests.exceptions.SSLError as errssl:
-
-            pass
-
-        except requests.exceptions.HTTPError as errh:
-
-            pass
-
-        except requests.exceptions.ConnectionError as errc:
-
-            pass
-
-        except requests.exceptions.ReadTimeout as to_err:
-
-            pass
-
-        except requests.exceptions.TooManyRedirects as tmr_err:
-
-            pass
-
-        except urllib3.exceptions.LocationParseError as lp_err:
-
-            pass
-
-        except requests.exceptions.InvalidURL as iu_err:
-
-            pass
-
-        except requests.exceptions.ChunkedEncodingError as chunk_err:
-
-            print(f'ChunkedEncodingError! {server}')
-            pass
-
-        except ray.exceptions.RaySystemError as ray_sys_error:
-
-            print(ray_sys_error) 
-            pass
-
-        else:
-
-            if is_nodeinfo:
-
-                if nodeinfo != '/api/v1/instance?':
-
-                    if nodeinfo != '/.well-known/x-nodeinfo2?':
-
-                        try:
-
-                            self.soft = nodeinfo_json['software']['name']
-                            self.soft = self.soft.lower()
-                            self.soft_version = nodeinfo_json['software']['version']
-                            self.users = nodeinfo_json.get('usage').get('users').get('total') or '0'
-                            if int(self.users) > 1000000:
-                                return
-                            self.mau = nodeinfo_json.get('usage').get('users').get('activeMonth') or 0
-
-                            self.alive = True
-
-                            self.db.write_api(server, self.soft, self.users, self.mau, self.alive, nodeinfo, self.soft_version)
-
-                            print(f"Server {server} ({self.soft} {self.soft_version}) is alive!")
-                            print('*********************************************************************')
-
-                            return (server, self.soft, self.soft_version, self.users, self.mau, self.alive)
-
-                        except:
-
-                            pass
-
-                    else:
-
-                        try:
-
-                            self.soft = nodeinfo_json['server']['software']
-                            self.soft = soft.lower()
-                            self.soft_version = nodeinfo_json['server']['version']
-                            self.users = nodeinfo_json['usage']['users']['total']
-                            if int(self.users) > 1000000:
-                                return
-                            self.mau = nodeinfo_json.get('usage').get('users').get('activeMonth') or 0
-
-                            self.alive = True
-
-                            if self.soft == 'socialhome':
-
-                                self.db.write_api(server, soft, users, alive, nodeinfo, soft_version)
-
-                                print('*********************************************************************')
-                                print(f"Server {server} ({self.soft} {self.soft_version}) is alive!")
-                                print('*********************************************************************')
-
-                            return
-
-                        except:
-
-                            pass
-
-                if self.soft == '' and nodeinfo == "/api/v1/instance?":
-
-                    self.soft = 'mastodon'
-
-                    try:
-
-                        self.users = nodeinfo_json['stats']['user_count']
-
-                        if int(self.users) > 1000000:
-
-                            return
-
-                    except:
-
-                        self.users = 0
-
-                    try:
-
-                        self.soft_version = nodeinfo_json['version']
-
-                    except:
-
-                        self.soft_version = 'unknown'
-
-                    alive = True
-
-                    self.db.write_api(server, self.soft, self.users, self.alive, nodeinfo, self.soft_version)
-
-                    print('*********************************************************************')
-                    print(f"Server {server} ({self.soft}) is alive!")
-
-                elif self.soft == 'zap' and nodeinfo == "/api/v1/instance?":
-
-                    self.soft = 'zap'
-                    self.users = nodeinfo_json['stats']['user_count']
-                    self.soft_version = nodeinfo_json['version']
-                    self.alive = True
-
-                    print(server, self.soft, self.users, self.alive, api)
-
-                    print('*********************************************************************')
-                    print(f"Server {server} ({self.soft}) is alive!")
-
-            else:
-
-                print(f'Server {server} is dead')
-                print('*********************************************************************')
-
-        return (server, self.soft, self.soft_version, self.users, self.mau, self.alive)
-

requirements.txt

@@ -1,8 +1,5 @@
-requests
-psycopg2-binary
-pytz
-ray
-Mastodon.py
-matplotlib
-pandas
-humanfriendly
+Mastodon.py>=1.5.1
+psycopg2-binary>=2.8.4
+aiohttp>=3.6.2
+aiodns>=2.0.0
+matplotlib>=3.3.4

setup.py

@@ -1,157 +1,188 @@
-import os
-import sys
-from datetime import datetime
-import pytz
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+import getpass
 from mastodon import Mastodon
 from mastodon.Mastodon import MastodonMalformedEventError, MastodonNetworkError, MastodonReadTimeout, MastodonAPIError, MastodonIllegalArgumentError
-import pdb
+import fileinput,re
+import os
+import sys
 
-class Setup():
+def create_dir():
+  if not os.path.exists('secrets'):
+    os.makedirs('secrets')
 
-    name = 'fediverse setup'
+def create_file():
+  if not os.path.exists('secrets/secrets.txt'):
+    with open('secrets/secrets.txt', 'w'): pass
+    print(secrets_filepath + " created!")
 
-    def __init__(self, config_file=None, mastodon_hostname=None, peers_api=None, user_agent=None, secrets_filepath=None, mastodon_app_token=None):
+def create_config():
+  if not os.path.exists('config'):
+    os.makedirs('config')
+  if not os.path.exists(config_filepath):
+    print(config_filepath + " created!")
+    with open('config/config.txt', 'w'): pass
 
-        self.config_file = "config/config.txt"
-        self.mastodon_hostname = self.__get_parameter("mastodon_hostname", self.config_file)
-        self.peers_api = '/api/v1/instance/peers?'
-        self.user_agent = {'User-agent': "fediverse's stats (fediverse@mastodont.cat)"}
+def write_params():
+  with open(secrets_filepath, 'a') as the_file:
+    print("Writing secrets parameter names to " + secrets_filepath)
+    the_file.write('uc_client_id: \n'+'uc_client_secret: \n'+'uc_access_token: \n')
 
-        self.secrets_filepath = 'secrets/secrets.txt'
+def write_config():
+  with open(config_filepath, 'a') as the_file:
+    the_file.write('mastodon_hostname: \n')
+    print("adding parameter name 'mastodon_hostname' to "+ config_filepath)
 
-        is_setup = self.__check_mastodon_setup(self)
+def read_client_lines(self):
+  client_path = 'app_clientcred.txt'
+  with open(client_path) as fp:
+    line = fp.readline()
+    cnt = 1
+    while line:
+       if cnt == 1:
+         print("Writing client id to " + secrets_filepath)
+         modify_file(secrets_filepath, "uc_client_id: ", value=line.rstrip())
+       elif cnt == 2:
+         print("Writing client secret to " + secrets_filepath)
+         modify_file(secrets_filepath, "uc_client_secret: ", value=line.rstrip())
+       line = fp.readline()
+       cnt += 1
 
-        if is_setup:
+def read_token_line(self):
+  token_path = 'app_usercred.txt'
+  with open(token_path) as fp:
+    line = fp.readline()
+    print("Writing access token to " + secrets_filepath)
+    modify_file(secrets_filepath, "uc_access_token: ", value=line.rstrip())
 
-            self.mastodon_app_token = self.__get_mastodon_parameter("mastodon_app_token", self.secrets_filepath)
+def read_config_line():
+  with open(config_filepath) as fp:
+    line = fp.readline()
+    modify_file(config_filepath, "mastodon_hostname: ", value=hostname)
 
-        else:
+def log_in():
+  error = 0
+  try:
+    global hostname
+    hostname = input("Enter Mastodon hostname: ")
+    user_name = input("User name, ex. user@" + hostname +"? ")
+    user_password = getpass.getpass("User password? ")
+    app_name = input("This app name? ")
+    Mastodon.create_app(app_name, scopes=["read","write"],
+      to_file="app_clientcred.txt", api_base_url=hostname)
+    mastodon = Mastodon(client_id = "app_clientcred.txt", api_base_url = hostname)
+    mastodon.log_in(
+       user_name,
+       user_password,
+       scopes = ["read", "write"],
+       to_file = "app_usercred.txt"
+    )
+  except MastodonIllegalArgumentError as i_error:
+    error = 1
+    if os.path.exists("secrets/secrets.txt"):
+      print("Removing secrets/secrets.txt file..")
+      os.remove("secrets/secrets.txt")
+    if os.path.exists("app_clientcred.txt"):
+      print("Removing app_clientcred.txt file..")
+      os.remove("app_clientcred.txt")
+    sys.exit(i_error)
+  except MastodonNetworkError as n_error:
+    error = 1
+    if os.path.exists("secrets/secrets.txt"):
+      print("Removing secrets/secrets.txt file..")
+      os.remove("secrets/secrets.txt")
+    if os.path.exists("app_clientcred.txt"):
+      print("Removing app_clientcred.txt file..")
+      os.remove("app_clientcred.txt")
+    sys.exit(n_error)
+  except MastodonReadTimeout as r_error:
+    error = 1
+    if os.path.exists("secrets/secrets.txt"):
+      print("Removing secrets/secrets.txt file..")
+      os.remove("secrets/secrets.txt")
+    if os.path.exists("app_clientcred.txt"):
+      print("Removing app_clientcred.txt file..")
+      os.remove("app_clientcred.txt")
+    sys.exit(r_error)
+  except MastodonAPIError as a_error:
+    error = 1
+    if os.path.exists("secrets/secrets.txt"):
+      print("Removing secrets/secrets.txt file..")
+      os.remove("secrets/secrets.txt")
+    if os.path.exists("app_clientcred.txt"):
+      print("Removing app_clientcred.txt file..")
+      os.remove("app_clientcred.txt")
+    sys.exit(a_error)
+  finally:
+    if error == 0:
 
-            self.mastodon_app_token = self.mastodon_setup(self)
+      create_dir()
+      create_file()
+      write_params()
+      client_path = 'app_clientcred.txt'
+      read_client_lines(client_path)
+      token_path = 'app_usercred.txt'
+      read_token_line(token_path)
+      if os.path.exists("app_clientcred.txt"):
+        print("Removing app_clientcred.txt temp file..")
+        os.remove("app_clientcred.txt")
+      if os.path.exists("app_usercred.txt"):
+        print("Removing app_usercred.txt temp file..")
+        os.remove("app_usercred.txt")
+      print("Secrets setup done!\n")
 
-    @staticmethod
-    def __check_mastodon_setup(self):
+def modify_file(file_name,pattern,value=""):
+    fh=fileinput.input(file_name,inplace=True)
+    for line in fh:
+        replacement=pattern + value
+        line=re.sub(pattern,replacement,line)
+        sys.stdout.write(line)
+    fh.close()
 
-        is_setup = False
+# Returns the parameter from the specified file
+def get_parameter( parameter, file_path ):
+    # Check if secrets file exists
+    if not os.path.isfile(file_path):
+        print("File %s not found, creating it."%file_path)
+        log_in()
 
-        if not os.path.isfile(self.secrets_filepath):
-            print(f"File {self.secrets_filepath} not found, running setup.")
-        else:
-            is_setup = True
+    # Find parameter in file
+    with open( file_path ) as f:
+        for line in f:
+            if line.startswith( parameter ):
+                return line.replace(parameter + ":", "").strip()
 
-        return is_setup
+    # Cannot find parameter, exit
+    print(file_path + " Missing parameter %s "%parameter)
+    sys.exit(0)
 
-    @staticmethod
-    def mastodon_setup(self):
+# Returns the parameter from the specified file
+def get_hostname( parameter, config_filepath ):
+    # Check if secrets file exists
+    if not os.path.isfile(config_filepath):
+        print("File %s not found, creating it."%config_filepath)
+        create_config()
 
-        if not os.path.exists('secrets'):
-            os.makedirs('secrets')
+    # Find parameter in file
+    with open( config_filepath ) as f:
+        for line in f:
+            if line.startswith( parameter ):
+                return line.replace(parameter + ":", "").strip()
 
-        self.mastodon_user = input("Mastodon user login? ")
-        self.mastodon_password = input("Mastodon user password? ")
-        self.app_name = 'fediverse'
+    # Cannot find parameter, exit
+    print(config_filepath + "  Missing parameter %s "%parameter)
+    write_config()
+    read_config_line()
+    print("setup done!")
+    sys.exit(0)
 
-        self.mastodon_app_token = self.mastodon_log_in()
+# Load secrets from secrets file
+secrets_filepath = "secrets/secrets.txt"
+uc_client_id     = get_parameter("uc_client_id",     secrets_filepath)
+uc_client_secret = get_parameter("uc_client_secret", secrets_filepath)
+uc_access_token  = get_parameter("uc_access_token",  secrets_filepath)
 
-        if not os.path.exists(self.secrets_filepath):
-            with open(self.secrets_filepath, 'w'): pass
-            print(f"{self.secrets_filepath} created!")
-
-        with open(self.secrets_filepath, 'a') as the_file:
-            print("Writing Mastodon parameters to " + self.secrets_filepath)
-            the_file.write(f'mastodon_app_token: {self.mastodon_app_token}')
-
-        return self.mastodon_app_token
-
-    def mastodon_log_in(self):
-
-        token = ''
-
-        try:
-
-            response = Mastodon.create_app(
-                self.app_name,
-                scopes=["read","write"],
-                to_file=None,
-                api_base_url=self.mastodon_hostname
-                )
-            client_id = response[0]
-            client_secret = response[1]
-
-            mastodon = Mastodon(client_id = client_id, client_secret = client_secret, api_base_url = self.mastodon_hostname)
-        
-            token = mastodon.log_in(
-                self.mastodon_user,
-                self.mastodon_password,
-                scopes = ["read", "write"],
-                to_file = None
-            )
-
-            print('Log in succesful!')
-
-        except MastodonIllegalArgumentError as i_error:
-
-            sys.stdout.write(f'\n{str(i_error)}\n')
-
-        except MastodonNetworkError as n_error:
-
-            sys.stdout.write(f'\n{str(n_error)}\n')
-
-        except MastodonReadTimeout as r_error:
-
-            sys.stdout.write(f'\n{str(r_error)}\n')
-
-        except MastodonAPIError as a_error:
-
-            sys.stdout.write(f'\n{str(a_error)}\n')
-
-        finally:
-
-            return token
-
-    def __get_parameter(self, parameter, config_file):
-
-        if not os.path.isfile(config_file):
-            print(f"File {config_file} not found..")
-
-            self.mastodon_hostname = input("\nMastodon hostname: ")
-
-            self.__create_config(self)
-            self.__write_config(self)
-
-        with open( self.config_file ) as f:
-            for line in f:
-                if line.startswith( parameter ):
-                    return line.replace(parameter + ":", "").strip()
-
-    def __get_mastodon_parameter(self, parameter, secrets_filepath):
-
-        if not os.path.isfile(secrets_filepath):
-            print(f"File {secrets_filepath} not found..")
-
-            self.sign_in()
-
-        with open( self.secrets_filepath ) as f:
-            for line in f:
-                if line.startswith( parameter ):
-                    return line.replace(parameter + ":", "").strip()
-
-    @staticmethod
-    def __create_config(self):
-
-        if not os.path.exists('config'):
-
-            os.makedirs('config')
-
-        if not os.path.exists(self.config_file):
-
-            print(self.config_file + " created!")
-            with open(self.config_file, 'w'): pass
-
-    @staticmethod
-    def __write_config(self):
-
-        with open(self.config_file, 'a') as the_file:
-
-            the_file.write(f'mastodon_hostname: {self.mastodon_hostname}')
-            print(f"adding parameters to {self.config_file}\n")
+# Load configuration from config file
+config_filepath = "config/config.txt"
+mastodon_hostname = get_hostname("mastodon_hostname", config_filepath)

uptime.py

@@ -1,44 +0,0 @@
-import time
-import os
-from datetime import datetime, timedelta
-import humanfriendly
-from setup import Setup
-from database import Database
-from mastodon import Mastodon
-
-# main
-
-if __name__ == '__main__':
-
-    setup = Setup()
-
-    mastodon = Mastodon(
-            access_token = setup.mastodon_app_token,
-            api_base_url= setup.mastodon_hostname
-            )
-
-    db = Database()
-
-    alive_servers, max_uptime, best_servers, software_lst, servers_lst = db.get_uptime()
-    
-    toot_text = '\nAlive servers: ' + str(alive_servers)
-    toot_text += '\n\n'
-    toot_text += f"Best #fediverse's server uptime:\n{humanfriendly.format_timespan(max_uptime)}"
-    toot_text += '\n'
-    toot_text += f'\nBest uptime servers:\n{str(best_servers)} ({str(round((best_servers*100)/alive_servers,2))}%)'
-    toot_text += '\n\n'
-    toot_text += 'Best uptime softwares & servers:\n'
-    i = 0
-    while i < len(software_lst):
-
-        soft_percent = db.get_percentage(servers_lst[i], software_lst[i])
-        toot_text += ':' + str(software_lst[i]) + ': ' + str(servers_lst[i]) + ' (' + str(soft_percent) + '%)\n'
-        
-        if len(toot_text) > 470:
-            break
-        i +=1
-
-    print("Tooting...")
-    print(toot_text)
-
-    mastodon.status_post(toot_text, in_reply_to_id=None)