Weekly Dev Update #59

Hey Y’all, 

This week we’ve mostly worked on bugs that were affecting Service Node operators as they transitioned from lokid 3.0.X to the 4.0.3 suite of Loki Service Node tools. We published a new release for the Loki Storage Server and Launcher, and we have also made good progress on transitioning Loki Messenger to the mainnet so it can take advantage of the ~550 Loki Storage Servers.

Loki Core


Lokinet

If you’re on our Discord you can catch Jeff, the lead developer of LLARP, live streaming as he codes at https://www.twitch.tv/uguu25519. He typically streams on Tuesday mornings, 9am – 12pm Eastern (US) time.

What’s going on this week with Lokinet: 

All of the pull requests for the next version are either merged, or reviewed and to be merged in the next day or so. We are spinning up a distributed “toy” network to test the stability and functionality of the codebase in anticipation of a public release sometime in the next week or two (testing dependent). We also have internally-working debian packages for various recent Debian and Ubuntu versions to allow easy installation of Lokinet on those systems, and plan to release these with the public release.

Changelog:

New/updated Pull Requests:


Loki Messenger Desktop 

Storage Server


Loki Launcher

What’s going on this week with Loki Launcher:

With the launch of the Hefty Heimdall hardfork, the Loki Storage Server is now active. This week, after tracking down a number of bug reports submitted in Telegram and Discord about unexpected crashes, we found the suspect behaviour and put in a work around into the 1.0.0 release of the launcher. We also added a couple of fixes as a few new eyes were on the code, and improved its accuracy of status and startup.

Changelog:

  • Fix Storage Server pipe that would lock up Storage Server
  • Fix Storage Server stderr handler typo
  • SIGHUP guard fix
  • Double check running pid
  • Use SIGTERM instead of SIGINT to stop processes
  • Handle socket write errors better
  • Test socket for connectivity in status
  • Clear stale pid and socket files
  • Move uncaught exception log into var_path
  • Make sure Storage Server is running before startup is successful
  • Change version for git checkouts to be the last committed revision

Loki Blocks Onion Explorer 

The Loki Block Explorer has been expanded to show a number of new things including checkpoints and their votes, and decommissioned or inactive nodes. 

Loki Messenger on Mobile (iOS and Android)

We are very close to an Android release of Loki Messenger, and are now testing it internally in the office.


Thanks,  

Kee 

Weekly Dev Update #58

Loki Network Development Update 58

Hey Y’all, 

The Loki 4.0.0 hardfork is fast approaching! It’s happening in approximately 24 hours, so if you haven’t upgraded your Service Node, Miner or Mining Pool, now is your chance. If you don’t update in time, you will be left on an alternate chain and won’t be able to talk to the majority of the network.

A full guide on how to update can be found here: https://loki.network/2019/07/12/hefty-heimdall-mandatory-upgrade-period/

Loki Core


Lokinet

If you’re on our Discord you might catch Jeff, the lead developer of LLARP, live streaming as he codes: https://www.twitch.tv/uguu25519.  He typically streams on Tuesday mornings, 9am-12pm Eastern (US) time.

What’s going on this week with Lokinet:

Lokinet is entering a feature freeze for an upcoming release to the public, and is undergoing heavy internal testing to see how the network performs under various types of load.  We don’t have a fixed release date yet — it will depend on how testing goes this week, but look for one soon. The last several weeks of development have fixed a myriad of issues, big and small, and we think Lokinet will be ready for public testing soon.  Hence, we have internally frozen the codebase* to not add anything new (just important fixes!) between now and the 0.5.0 release.

* There is one exception; see below.

Changelog:

New Pull Requests:


Loki Messenger Desktop 

Storage Server


Loki Blocks Onion Explorer 

The Loki block explorer has been expanded to show a number of new things including checkpoints and their votes, and decommissioned or inactive nodes. 


Messenger Mobile (iOS and Android)


Thanks,  

Kee 

Hefty Heimdall Changes for Service Node Operators

There are a number of new and changing rules being implemented in the Loki Hefty Heimdall hardfork, and Service Node operators should make sure they’re aware of the new requirements.

The following changes will be implemented on July 24 at block height 321,467.

Changing Rules:

  • We have relaxed the Service Node deregistration rules in order to be more lenient on Service Nodes, particularly those that have previously demonstrated quality performance. 

New Rules

  • All Service Nodes must now run the Loki Storage Server to receive rewards.
  • All Service Nodes must now have an accessible Public IP address and must open ports for the Loki Storage Server.
  • There is now a penalty for Service Nodes that send their uptime proofs from different IP addresses.

More Detail:

Relaxed Deregistration: 

After reviewing feedback from Service Node operators on Discord and Github over the past 6 months, the Loki team proposed a number of relaxations (included in Hefty Heimdall) with regard to how Service Nodes are deregistered when they fail to meet the expected requirements. 

A few basic changes were made to implement the new system, including the introduction of a credit scheme and decommission/recommission transactions. The basic scheme is detailed below:

  • Each Service Node starts with 2 hours of credit.
  • Each Service Node earns 0.8 hours of credit per day of uninterrupted operation up to a maximum of 24 hours.
  • If a Service Node has been assessed to be no longer meeting the standards of the network (for example, not submitting uptime proofs) the quorum looks to see if they have any credit.
  • If the Service Node has 2 hours or more in credit, it will be decommissioned (instead of deregistered) for the amount of time it has in credit.
  • When a Service Node is decommissioned, its funds are not locked for 30 days but it is removed from the rewards pool and cannot participate in normal network functions.
  • If the Service Node starts sending uptime proofs again during the decommission period, the current quorum will detect this and submit a recommission transaction which will reset the Service Node’s credit balance to zero, and insert the decommissioned Service Node back to the bottom of the rewards list.
  • If, during the decommission period, the Service Node’s credit runs out before it can successfully submit an uptime proof, it will be deregistered.

TL;DR Service Nodes now have a longer grace period which grows the longer your Service Node is up and performing well. Service Nodes that have been running without interruption for 30 days will now have 24 hours where they can be offline before they are deregistered and your funds are locked for 30 days.

Read the full details of the relaxed deregistration rules here: https://lokidocs.com/ServiceNodes/DeregistrationRules/


Loki Storage Server:

The Loki Storage Server is an application that exposes a public endpoint for Loki Messenger users to be able to store and retrieve messages on your Service Node. It is a necessary requirement for Loki Messenger.

Hefty Heimdall versions of lokid running in Service Node mode will look for a running Loki Storage Server on your machine. If lokid does not find a running Storage Server, it will refuse to start. Lokid will also periodically check to see if the Loki Storage Server is still running – if it isn’t, lokid will stop broadcasting uptime proofs.  

You can easily manage lokid and the Loki Storage Server with the Loki Launcher, which sets up the required utilities. If you are an experienced system administrator, you can manage these utilities yourself – though we recommend you test your setup on testnet.

TL;DR You need to run the Loki Storage Server alongside lokid. Loki Launcher will do this automatically, otherwise binaries can be found here: https://github.com/loki-project/loki-storage-server


Public IP address and Open Ports:

All Service Nodes must now have a public IP address and open ports for lokid P2P and the Loki Storage Server. The default port for mainnet lokid P2P is 22022, and for the Loki Storage Server the default port is 23023.

If you are currently running your Service Node without any open ports or behind NAT, you will need to look into creating port forwarding rules in your router, and/or opening these ports in your firewall. If you are running a custom setup, you can change your default ports for Loki Storage Server and lokid P2P communications.

TL;DR You need to open ports in your firewall or router and have a public IP address.


IP Address Change Penalty: 

We understand that many Service Node operators in the community are running backup servers which submit uptime proofs for their Service Node. Most operators have set up these backup servers by running another lokid with the –service-node flags on a seperate IP, but with the same Service Node key as their primary Service Node.

This setup creates a race between the two Service Nodes, which compete to send out the first uptime proof. Depending on the precise timing of when the separate lokids submit their uptime proofs, the relationship between the two Service Nodes can change. You may find one is master for a while, and then it switches. You may find the two Services Nodes swap on every announcement. 

This race condition will be a problem in Hefty Heimdall because of the inclusion of the Service Node IP address in each uptime proof. Every time the server sending the uptime proof “wins”, it changes the IP on which the network tries to reach the Service Node’s Storage Server, and of course, the backup won’t have the same messages stored as the primary server.

Without modifying the Storage Server code to create a syncing channel between a master and backup Service Node, this problem is difficult to solve. Instead, the Loki team has implemented a punishment for Service Nodes that submit uptime proofs from different IP addresses. Each time your IP address changes, you will be dropped to the bottom of the rewards list losing your previous position. 

From talking to Service Node operators, we found that one of the most common reasons they ran Service Node backups was because they didn’t have enough time to respond to outages on their Service Nodes. By addressing the core issue – which was the lack of forgiveness in the deregistration rules – we think there will be far fewer people wanting to run backup Service Nodes. 

Although this change does not explicitly prevent the running of backup Service Nodes, Service Node operators who choose to run backup servers should ensure that they only send uptime proofs from the backup Service Node when it’s clear that the master server is actually down, rather than just setting up two Service Nodes and having both submit proofs at competing times.

TL;DR Running two Loki Service Nodes with the same SN key on two machines with different IP addresses will likely lead to your Service Node being dropped to the bottom of the rewards list regularly.


We are really excited to see these changes go live on mainnet on July 24. The Loki team and Loki community are always looking for ways to improve the stability and usefulness of the Service Node network, while maintaining a simple user experience so the Service Node network can continue to grow.  

Weekly Dev Update #57

Hey Y’all, 

This week was particularly busy as we worked towards the final binaries release for the mandatory upgrade period. Thank you to everyone who jumped on testnet and set up nodes – with your help we were able to identify a host of bugs which were fixed in our final 4.0.3 release. 

If you are a Service Node operator you should upgrade your node to version 4.0.3. Instructions on how to do this can be found here: https://lokidocs.com/ServiceNodes/SNFullGuide/#updating-your-binaries 

Loki Core


Loki Launcher

The Loki Launcher is a node JS package that will allow for the independent management of all the components required to run a full Service Node. This includes managing Lokinet, lokid, the Loki Storage Server and any other future applications we require. When Loki Service Nodes begin to route data and store messages for Lokinet and Loki Messenger, we’ll recommend that every single Service Node run the Loki Launcher.

What’s going on this week with Loki Launcher:

We released two versions of the Launcher this week for the 4.0.3 release. There were a lot of fixes for various crashes, but also support to be able to run versions 3 and up of lokid on mainnet. We also added extra startup checks to improve our status accuracy and made some minor user experience improvements.

Changelog:

  • Prequal additional configuration system for Storage Server
  • Update prequal ports to test
  • Rename port names, adjust prequal output
  • Abort launcher start up if Storage Server port is not open to the public
  • Handle “port is in use” error handling better
  • Store stdout/stderr for launcher when backgrounding
  • Add *-server-port-check args
  • Wait for blockchain-rpc port to be open before considering start up a success
  • Temporarily collect Storage Server startup info for 10s in case of problems with output
  • Report if the launcher was already stopped when stopping internally
  • Make sure certain modes are only run with root
  • Activate Storage Server based on blockchain binary version
  • Only pass version 4  parameters when blockchain binary version 4 or above
  • Reminders to restart loki-launcher if mode has stopped it
  • Strip out launcher arguments from passing all the way to blockchain startup
  • Remove duplicate Storage Server port check when backgrounding
  • Put try/catch guard around process check to prevent intermittent launcher crashes
  • When running port test submit a disconnect message when done
  • Add unhandled exception logger
  • Improve HUP information format
  • Process blockchain stderr
  • Make sure blockchain rpc server is up before starting network/storage
  • Change Storage Server default port from 8080 to 23023
  • Up retries for getPublicIPv4
  • Untangle retry counter with various repos in download-binaries

Github Pulse: Excluding merges, 2 authors have pushed 41 commits to master and 41 commits to all branches. On master, 14 files have changed and there have been 590 additions and 196 deletions


Lokinet

If you’re on our Discord you might catch Jeff or Ryan, the developers of LLARP, live streaming as they code: https://www.twitch.tv/uguu25519, https://www.twitch.tv/neuroscr

What’s going on this week with Lokinet:

The configuration refactor finally reached a state close to stable. There were lots of MacOS and FreeBSD fixes, and a huge internal change to using IPv6 on tunnel adapters to allow more connections to a hidden service or client.

Changelog:

Pull Requests:


Loki Messenger Desktop 

Storage Server


Loki Blocks Onion Explorer 

The Loki block explorer has been expanded to show a number of new things, including checkpoints and their votes and decommissioned or inactive nodes. 

  • Updated RPC changes around quorum state calls in 4.0.2 (replaced batch quorum calls with new quorum states call). 
  • Add checkpoint quorum display to quorum states page and to main index. 
  • Add display of a single obligations or checkpoint quorum, linked where appropriate.
  • Cut off quorum display after 20 nodes with a “+ 7 more ↪” that links to the full obligations quorum details.
  • Add a list of last three checkpoints to the main page (and a link to a page with up to 50 fully displayed). https://github.com/loki-project/loki-onion-blockchain-explorer/pull/1

Messenger Mobile (iOS and Android)


Thanks,  

Kee