Development 3botdeployer (#1441)

* Add location choice

* Block the chatflow until the farms are ready

* Add location policy

* Fetch farm name early

* Fix pool selection when forcing specific node

* Use drop down menu instead of single choice

* Create wallet on entry

* working on adding 3bot status

* Create a new wallet on 3bot creation

* Rename the wallet to mainnet_wallet

* Move wallet creation to threebot start

* Log wallet creation errors and continue starting

* Don't discriminate between self hosted and chatflow hosted 3Bots

* update backup path related to restic server

* Deploy on the new identity

* add flavors for deployer

* override the flavors

* Create 3bot on new pool and new managed network

* add destroy, delete and restart actions

* add /root/.ssh to restic backup paths

* add the deployer actions depending on the status

* update methods with identity_name wip

* separate 3bot deployers in separate component

* refactor models dir

* fix saved subdomain id

* fix instance name

* working on the frontend for listin the 3bots

* use mainnet default

* fixing and issue related to back /.ssh path

* threebot listing wip

* check for workload container type in building info

* remove debugging prints

* expired pools have empty_at at max int64

* add stop and delete threebot solutions

* check if metadata exists before trying to decrypt it

* fix typos

* set owner key in networks metadata updated by the admin dashboard

* move threebot config validation to a separate method

* remove unused statement

* address comments

* deploy on mainnet

* Development 3botdeployer escalation emails (#1478)

* Add escalation emails button in developer options in fron-end and update the back-end

* Add escalation email front-end and back-end

* Handle the None stat in config manager related to escalating emails

* fix invalid import

* use management as name for networks 3bot deployment

* pass identity to dry_run context

* use new listing util in threebot chatflow

* fix network name

* return expiration

* Update front-end and back end to use the new methods in listing the threebot states

* add start method in utils

* Restart 3Bot wip

* Adds correct actions for stop / destroy threebot for threebot deployer

Signed-off-by: Ashraf Fouda <[email protected]>

* Fix steps missing comma

* Add final step

* Add some hardening for threebot deletion and stopping

* clean up threebot identity

* updated frontend view

* delete only deployed workloads

* add change size and location chatflows

* receive 3bot name from params

* clean up unused identity

* fix log config

* fix log config

* fix message

* remove extend

* clear workloads in deployment_context before deleting threebot identity

* show the correct empty at in blocked nodes

* one table for all workloads

* Add state filter in deployed 3bots UI

* show bell icon and extend message and remove expiration column

* update quick start guide

* update quick start guide

* change expiration icon

* show href in info if the threebot is running

* remove not needed inof when threebot is not running

* fix error message

* Create wallets when deploying new threebot

* remove unused actor methods

* override the flist with latest flist on redeployments

* override the flist with latest flist on redeployments

* fix api url and remove deprecated apis

* return status 400 instead of raising exception

* fix cancel solution for other identities

* update branch in redeployments

* open url in another tab

* pass identity name to metadata encryption & decryption in delete_access

* use root instead of homedir

* change variable name

* change ssh dir path based on the current user running 3bot server

* fix message

* revert ssh dir

* respect over provisioning in farm capacity check

* remove extend test

* use identity name when decrypting metadata

* Development 3botdeployer autoextend pool (#1526)

auto-extend pools

* Use std instead of default identity for the main wallet

* update threebot tests

* remove conflicts

* remove recover and extend tests

* Update email server config in admin dashboard vue to include tooltips

* add threebot start & change size gedis patches

* add threebot change location patch

* add threebot change location patch

* init test cases for threebot deployer

* Make development as the default branch for threebot deployer

Co-authored-by: Omar Elawady <[email protected]>
Co-authored-by: Rafy Benjamin <[email protected]>
Co-authored-by: m-motawea <[email protected]>
Co-authored-by: RafyAmgadBenjamin <[email protected]>
Co-authored-by: Ashraf Fouda <[email protected]>
Co-authored-by: ranatrk <[email protected]>
Co-authored-by: Waleed <[email protected]>

Author: 7 people

jumpscale/entry_points/threebot.py

@@ -11,6 +11,7 @@
 from jumpscale.loader import j
 from jumpscale.threesdk.identitymanager import IdentityManager
 from jumpscale.sals.nginx.nginx import PORTS
+from jumpscale.packages.admin.actors.wallet import Wallet
 
 SERVICES_PORTS = {"nginx": 8999, "nginx_http": 80, "nginx_https": 443, "gedis": 16000}
 
@@ -80,6 +81,9 @@ def start(identity=None, background=False, local=False, development=False, domai
         identity (str, optional): threebot name. Defaults to None.
         explorer (str, optional): which explorer network to use: mainnet, testnet, devnet. Defaults to None.
     """
+    if j.config.get("ANNOUNCED") is None:
+        j.config.set("ANNOUNCED", False)
+    create_wallets_if_not_exists()
     check_for_bins()
     PORTS.init_default_ports(local)
     SERVICES_PORTS["nginx_http"] = PORTS.HTTP
@@ -227,7 +231,7 @@ def clean(all=False):
                 print(f"exception was {e} for debugging")
 
             answer = j.tools.console.ask_yes_no(f"Do you want to remove {config_root} ? ")
-            if answer=="y":
+            if answer == "y":
                 j.sals.fs.rmtree(config_root)
                 print("Previous configuration is deleted.")
 
@@ -236,12 +240,44 @@ def clean(all=False):
             print(f"exception for debugging {e}")
 
 
-
 @click.group()
 def cli():
     pass
 
 
+def have_wallets():
+    wallets = j.clients.stellar.list_all()
+    test, main = False, False
+    for wallet_name in wallets:
+        wallet = j.clients.stellar.get(wallet_name)
+        if wallet.network.value == "TEST":
+            test = True
+        elif wallet.network.value == "STD":
+            main = True
+    return test, main
+
+
+def create_test_wallet(wallet_name):
+    try:
+        j.clients.stellar.create_testnet_funded_wallet(wallet_name)
+    except Exception as e:
+        j.logger.error(str(e))
+
+
+def create_main_wallet(wallet_name):
+    wallet_actor = Wallet()
+    try:
+        wallet_actor.create_wallet(wallet_name, "STD")
+    except Exception as e:
+        j.logger.error(str(e))
+
+
+def create_wallets_if_not_exists():
+    test, main = have_wallets()
+    if not test:
+        create_test_wallet("test")
+    if not main:
+        create_main_wallet("main")
 
 
 cli.add_command(start)

jumpscale/packages/admin/actors/admin.py

@@ -185,32 +185,46 @@ def get_developer_options(self) -> str:
         test_cert = j.core.config.set_default("TEST_CERT", False)
         over_provision = j.core.config.set_default("OVER_PROVISIONING", False)
         explorer_logs = j.core.config.set_default("EXPLORER_LOGS", False)
+        escalation_emails = j.core.config.set_default("ESCALATION_EMAILS_ENABLED", False)
+        auto_extend_pools = j.core.config.set_default("AUTO_EXTEND_POOLS_ENABLED", False)
         sort_nodes_by_sru = j.core.config.set_default("SORT_NODES_BY_SRU", False)
         return j.data.serializers.json.dumps(
             {
                 "data": {
                     "test_cert": test_cert,
                     "over_provision": over_provision,
                     "explorer_logs": explorer_logs,
+                    "escalation_emails": escalation_emails,
+                    "auto_extend_pools": auto_extend_pools,
                     "sort_nodes_by_sru": sort_nodes_by_sru,
                 }
             }
         )
 
     @actor_method
     def set_developer_options(
-        self, test_cert: bool, over_provision: bool, explorer_logs: bool, sort_nodes_by_sru: bool
+        self,
+        test_cert: bool,
+        over_provision: bool,
+        explorer_logs: bool,
+        sort_nodes_by_sru: bool,
+        escalation_emails: bool,
+        auto_extend_pools: bool,
     ) -> str:
         j.core.config.set("TEST_CERT", test_cert)
         j.core.config.set("OVER_PROVISIONING", over_provision)
         j.core.config.set("EXPLORER_LOGS", explorer_logs)
+        j.core.config.set("ESCALATION_EMAILS_ENABLED", escalation_emails)
+        j.core.config.set("AUTO_EXTEND_POOLS_ENABLED", auto_extend_pools)
         j.core.config.set("SORT_NODES_BY_SRU", sort_nodes_by_sru)
         return j.data.serializers.json.dumps(
             {
                 "data": {
                     "test_cert": test_cert,
                     "over_provision": over_provision,
                     "explorer_logs": explorer_logs,
+                    "escalation_emails": escalation_emails,
+                    "auto_extend_pools": auto_extend_pools,
                     "sort_nodes_by_sru": sort_nodes_by_sru,
                 }
             }
@@ -222,6 +236,42 @@ def clear_blocked_nodes(self) -> str:
         return j.data.serializers.json.dumps({"data": "blocked nodes got cleared successfully."})
 
     @actor_method
+    def get_email_server_config(self) -> str:
+        email_server_config = j.core.config.get("EMAIL_SERVER_CONFIG", {})
+        email_server_config.setdefault("host", "")
+        email_server_config.setdefault("port", "")
+        email_server_config.setdefault("username", "")
+        email_server_config.setdefault("password", "")
+        return j.data.serializers.json.dumps({"data": email_server_config})
+
+    @actor_method
+    def set_email_server_config(self, host="", port="", username="", password="") -> str:
+        email_server_config = j.core.config.get("EMAIL_SERVER_CONFIG", {})
+        email_server_config = {"host": host, "port": port, "username": username, "password": password}
+        j.core.config.set("EMAIL_SERVER_CONFIG", email_server_config)
+        return j.data.serializers.json.dumps({"data": email_server_config})
+
+    @actor_method
+    def list_escalation_emails(self) -> str:
+        escalation_emails = j.core.config.get("ESCALATION_EMAILS", [])
+        return j.data.serializers.json.dumps({"data": escalation_emails})
+
+    @actor_method
+    def add_escalation_email(self, email) -> str:
+        escalation_emails = j.core.config.get("ESCALATION_EMAILS", [])
+        if email not in escalation_emails:
+            escalation_emails.append(email)
+            j.core.config.set("ESCALATION_EMAILS", escalation_emails)
+        return j.data.serializers.json.dumps({"data": escalation_emails})
+
+    @actor_method
+    def delete_escalation_email(self, email) -> str:
+        escalation_emails = j.core.config.get("ESCALATION_EMAILS", [])
+        if email in escalation_emails:
+            escalation_emails.remove(email)
+            j.core.config.set("ESCALATION_EMAILS", escalation_emails)
+        return j.data.serializers.json.dumps({"data": escalation_emails})
+
     def get_notifications(self) -> str:
         notifications = []
         if j.tools.notificationsqueue.count() >= 10:

jumpscale/packages/admin/actors/wallet.py

@@ -5,11 +5,13 @@
 
 class Wallet(BaseActor):
     @actor_method
-    def create_wallet(self, name: str) -> str:
+    def create_wallet(self, name: str, wallettype: str = None) -> str:
         explorer = j.core.identity.me.explorer
-        wallettype = "STD"
-        if "testnet" in explorer.url or "devnet" in explorer.url:
-            wallettype = "TEST"
+        if wallettype is None:
+            if "testnet" in explorer.url or "devnet" in explorer.url:
+                wallettype = "TEST"
+            else:
+                wallettype = "STD"
 
         if j.clients.stellar.find(name):
             raise j.exceptions.Value(f"Wallet {name} already exists")

jumpscale/packages/admin/bottle/admin.py

@@ -8,6 +8,7 @@
 
 app = Bottle()
 
+
 @app.route("/api/allowed", method="GET")
 @login_required
 def allowed():
@@ -44,4 +45,23 @@ def accept():
         )
 
 
+@app.route("/api/announced", method="GET")
+@login_required
+def announced():
+    result = bool(j.config.get("ANNOUNCED"))
+
+    return HTTPResponse(
+        j.data.serializers.json.dumps({"announced": result}), status=200, headers={"Content-Type": "application/json"}
+    )
+
+
+@app.route("/api/announce", method="GET")
+@login_required
+def announce():
+    j.config.set("ANNOUNCED", True)
+    return HTTPResponse(
+        j.data.serializers.json.dumps({"announced": True}), status=200, headers={"Content-Type": "application/json"}
+    )
+
+
 app = SessionMiddleware(app, SESSION_OPTS)

jumpscale/packages/admin/frontend/App.vue

@@ -136,6 +136,33 @@
       <identities v-model="dialogs.identity"></identities>
       <popup></popup>
     </v-main>
+     <v-dialog
+      v-model="announcement_dialog"
+      persistent
+      max-width="500"
+    >
+
+      <v-card>
+        <v-card-title class="headline">
+          Quick start guide
+        </v-card-title>
+        <v-card-text>
+        We've created two wallets for you; main and test. These are to be used for mainnet and testnet respectively. Make sure your wallets are funded to extend your 3Bots before they expire.
+        <br />
+        Please visit the <a href="https://manual.threefold.io">manual</a> for more information.
+        </v-card-text>
+        <v-card-actions>
+          <v-spacer></v-spacer>
+          <v-btn
+            color="green darken-1"
+            text
+            @click="announced = true"
+          >
+            Ok
+          </v-btn>
+        </v-card-actions>
+      </v-card>
+    </v-dialog>
   </v-app>
 </template>
 
@@ -171,14 +198,17 @@ module.exports = {
       dialogs: {
         identity: false,
       },
+      announced: true
     };
   },
   components: {
     identities: httpVueLoader("./Identity.vue"),
   },
-  computed: {},
   methods: {},
   computed: {
+    announcement_dialog() {
+      return !this.announced
+    },
     pages() {
       return this.$router.options.routes.filter((page) => {
         return page.meta.listed;
@@ -197,6 +227,13 @@ module.exports = {
         this.user = response.data;
       });
     },
+    getAnnouncementStatus() {
+      this.$api.announcement.announced().then((response) => {
+        console.log(response.data)
+        this.announced = response.data["announced"];
+        this.$api.announcement.announce();
+      });
+    },
     getIdentity() {
       this.$api.identity.get().then((response) => {
         this.identity = JSON.parse(response.data);
@@ -237,6 +274,7 @@ module.exports = {
     this.checkDarkMode();
     this.getIdentity();
     this.getCurrentUser();
+    this.getAnnouncementStatus();
     this.setTimeLocal();
     this.clockInterval = setInterval(() => {
       this.setTimeLocal();

jumpscale/packages/admin/frontend/api.js

@@ -181,12 +181,12 @@ const apiClient = {
                 url: `${baseURL}/admin/get_developer_options`
             })
         },
-        setDeveloperOptions: (testCert, overProvision, explorerLogs, sortNodesBySRU) => {
+        setDeveloperOptions: (testCert, overProvision, explorerLogs, escalationEmails, autoExtendPools, sortNodesBySRU) => {
             return axios({
                 url: `${baseURL}/admin/set_developer_options`,
                 method: "post",
                 headers: { 'Content-Type': 'application/json' },
-                data: { test_cert: testCert, over_provision: overProvision, explorer_logs: explorerLogs, sort_nodes_by_sru: sortNodesBySRU }
+                data: { test_cert: testCert, over_provision: overProvision, explorer_logs: explorerLogs, sort_nodes_by_sru: sortNodesBySRU, escalation_emails: escalationEmails, auto_extend_pools: autoExtendPools }
             })
         },
         clearBlockedNodes: () => {
@@ -205,6 +205,50 @@ const apiClient = {
             })
         }
     },
+    emailServerConfig: {
+        get: () => {
+            return axios({
+                url: `${baseURL}/admin/get_email_server_config`
+            })
+        },
+        set: (host, port, username, password) => {
+            return axios({
+                url: `${baseURL}/admin/set_email_server_config`,
+                method: "post",
+                headers: { "Content-Type": "application/json" },
+                data: {
+                    host: host,
+                    port: port,
+                    username: username,
+                    password: password
+                }
+            })
+        },
+    },
+    escalationEmails: {
+        list: () => {
+            return axios({
+                url: `${baseURL}/admin/list_escalation_emails`
+            })
+        },
+        add: (email) => {
+            return axios({
+                url: `${baseURL}/admin/add_escalation_email`,
+                method: "post",
+                headers: { 'Content-Type': 'application/json' },
+                data: { email: email }
+            })
+        },
+        delete: (email) => {
+            return axios({
+                url: `${baseURL}/admin/delete_escalation_email`,
+                method: "post",
+                headers: { 'Content-Type': 'application/json' },
+                data: { email: email }
+            })
+        }
+
+    },
     explorers: {
         get: () => {
             return axios({
@@ -534,5 +578,19 @@ const apiClient = {
                 method: "get"
             })
         },
+    },
+    announcement: {
+        announced: () => {
+            return axios({
+                url: `/admin/api/announced`,
+                method: "get"
+            })
+        },
+        announce: () => {
+            return axios({
+                url: `/admin/api/announce`,
+                method: "get"
+            })
+        },
     }
 }