Case study – Fitel Network S.L.

Fitel Networks is a company from Catalunya, Spain. The company has had its own wireless network for 2 years with over 1,000 subscribers connected.

 

 

 At the end of 2017, the Fitel team started to deploy fibre optics. Fibre was installed in the main urban locations of the city of Lloret de Mar. The topology that the technical management team decided to use is FTTH and FTTB, with GPON technology. Currently, the network is designed to connect over 10,000 fibre ONT devices. Services that are provided to end-users include VOIP and Mobile connections. The company connects companies and residential customers that pay on a monthly basis (recurring) and has a number of customers who come to Lloret de Mar for a vacation and pay for services in a prepaid mode.

 The Fitel Network was optimized by deploying the Splynx ISP billing system, redesigning the core network to allow 10G throughput, registering AS and IPv4 address space and deploying BGP services for their customers.

 

“Many hours of work, some days without sleep, all efforts are rewarded by the core of Fitel Network, which is prepared to compete with the big operators in IP traffic of 10G, implementation of BGP, OSPF, PPPOE servers, routing, implementation of security policies, and also the implementation of the software for ISP. A special thanks to SPLYNX team, Alex Vishnyakov and Paul Gerhardt for their dedication and for believing in this great project.”Erwin Cárdenas Barrera – General Director & CEO of Fitel Network S.L.

 

Let us describe how it started and the whole process

In the middle of 2018, Fitel Networks requested our team to install and deploy Splynx billing software. During installation and implementation, we noticed that the main MikroTik CCR router was having some strange issues. PPPoE users were continuously disconnecting, packets were lost and router behavior was unstable. Fitel’s technical officer Paul Gerhardt asked us to investigate if the problem was caused by the Splynx software, but we knew that Splynx can reconnect PPPoE sessions only in one case – when an administrator sends COA or the ‘kill session’ command. Our engineers were aware of this and started to analyze the network design.

 The company had Huawei MA5608 OLT equipment installed, which worked in bridge mode, terminating all PPPoE connections on a MikroTik CCR1036 router. This router was performing the role of PPPoE concentrator and NAT server. When the network had only wireless customers at around 800 Mbps, the router was performing well, but when the first few hundred GPON customers were connected, the router’s traffic reached 1.4 Gbps and the issues started.

 The main issue was that once a day at around 4pm, several customers reconnected their PPPoE sessions, which caused the CPU activity to jump from 30% to 100%, and, as a result, the router stopped responding to new PPPoE requests. Also, it started to disconnect dozens of already-connected PPPoE customers.

 A short investigation from our team showed that NAT connections tracking together with a large number of PPPoE connections and hence high traffic gives this issue. We offered Fitel Networks our service of network redesign and optimization.

 The initial topology looked as in the picture provided below:

  1. Wireless network. All users connect through the network on L2 bridges up to a CCR router. Customers have UBNT or MikroTik CPE equipment that works as a transparent bridge. Each customer has a TP-Link or MikroTik router installed on the premises which establish a PPPoE connection and also connects end users’ devices to their home Wi-Fi networks.
  2. Fibre network. A Huawei MA5608 router is connected to a MikroTik CCR router with two Gigabit interfaces. Interfaces are bonded using LACP. The company planned to migrate to 10G interfaces and was waiting for the module car
  3. MikroTik CCR1036. The router was connected via a 10G port to the uplink with PPPoE and NAT server running on it.

 

Our team has experience with designing ISP networks for over 10G loads. We have built and optimized several wireless and fibre networks and it is one of the services that a customer can get from us when deploying Splynx ISP software. The main point of the whole optimization was to move out the load from one main MikroTik router and share it between different equipment. There was also another reason – to change the MikroTik router that is CPU routing-based to some hardware accelerated router, for example, Juniper, Alcatel or Cisco.

Fitel Network’s IT team has MikroTik knowledge and, as such, changing the platform was very painless. We decided to take more than one MikroTik router, add switches to have L2 redundant topology, and share the load between different equipment.

The result of the redesign and optimization is presented in the picture below:

Logical topology created in visualization software:

 

Equipment that was installed

A MikroTik CCR1036 router acts as the BGP and NAT router

MikroTik CRS317 16 ports SFP+ – core switch, are connected to uplink 10G ports, a BGP router and all three PPPoE routers are also connected via 10G ports. Please note that one same switch is installed just close to the first one. The second switch acts as a cold backup. This means it has exactly the same configuration as the core switch and the same amount of ports, ready to swap in case of failure of the first switch.

3x MikroTik CCR1036 routers which work as PPPoE servers where fibre and wireless clients are connected.

MikroTik CRS328 24 ports + 4 SFP ports, which acts as a distribution switch. To this switch are connected GPON OLT and links coming from wireless towers. Also, PPPoE router downlinks are plugged-in to this switch.

A BGP router announces public prefixes to the Internet and routes traffic between two Internet providers. Uplinks are 10GB lines. Each PPPoE router has its own private 172.16.x.x network from which we assign IP addresses to private and prepaid customers. It means that on the BGP router there are three installed static routes to these networks – each network to a proper PPPoE router.

PPPoE servers are connected to the same switch on the same VLANs, which means that users from fibre and wireless networks can connect to one of three PPPoE servers at the same time. The advantage of this approach is scalability and failover. If one router fails, customers can automatically reconnect to the two remaining PPPoE routers.

PPPoE routers are linked with Splynx Radius server. When a customer connects to router1, he gets the IP from the pool, which is dedicated for this certain router.

Another story is with public IP addresses. In this case, customers have the static IP address, which means that physically they can connect to any PPPoE server.

To know where the BGP router should send traffic, we have activated OSPF on three links – between each PPPoE concentrator and the BGP router. PPPoE customers IPs are redistributed to OSPF as connected routes as soon as they connect to the Internet. It is important to set the routing filters correctly – allowing only public IPs in a routing table with /32 routes, and all private IPs are not redistributed to the OSPF.

Download case study Fitel Network&Splynx case study

HTTPS/SSL installation

Here in Splynx we are working very intensively on configuration part of the system. Many things that were available for configuration over CLI we have put to the Splynx GUI configuration. One of such examples is HTTLS activation and SSL certificate installation.

You should always protect Splynx server with HTTPS, even if it’s used inside the ISP network. HTTPS and SSL provides critical security and data integrity your server and your user’s personal information.

From Splynx release 2.1,the HTTPS/SSL option can be found under Config -> Main -> HTTPS / SSL

Splynx allows you to import your existing certificate, that was verified by official authority. In this case administrator should upload the SSL private key and Certificate.

In case when you don’t have certificate, you can request the Let’s encrypt free certificate via Splynx configuration. Click on the Let’s Enrypt and save. Important is to enter the domain name or subdomain name that is linked to your Splynx server (point A entry in domain settings to Splynx server IP address)

Now you are two steps far from getting SSL ready and running – click Validate to obtain SSL certificate and then Activate to run the HTTPS. As a result you should see all status messages with green highlighting :

From now, Splynx runs on Secure connection. Let’s Encypt is a trusted free, automated, and open Certificate Authority.


Should you have any questions regarding SSL/HTTPS installation or further information is needed, please contact us or schedule a call with our engineer.

Splynx GDPR compliance

From May 25th 2018, the EU’s General Data Protection Regulations (GDPR) came into effect.

As part of our commitment to transparency, and in preparation for the new data protection laws, we’ve updated our Software solution. We encourage you to read the GDPR Splynx compliancy document in full and to contact us if you have any questions.

Splynx is a management software for Internet Service Providers (referred to as ISP in the following text). It means that our company doesn’t work directly with data of individuals.

However, our clients operate in the B2C market and use our software to store their customers’ data. In this situation, Splynx s.r.o. works as a solution provider. The areas and situations described below are when Splynx software works with the data of individuals and how our software complies with GDPR regulations.

Storing personal information

-> Storing data of individuals in general

Splynx software is installed in the premises of an ISP company. In this case. all data is physically stored in the office or datacenter of our client. This means that our client is in charge of the physical security aspects of the server where the software runs.

Basic information that is stored in a Splynx database about customers is not sensitive. Data such as a customer’s address, phone number, passport/ID number or bank account are requested by ISP based on the customer’s agreement. For example, the ISP cannot provide services without being able to contact the customer and is not able to receive payments if it does not have the customer’s bank details. Technical information regarding a customer’s data tariff plan and a customer’s IP/MAC address is needed by the ISP to provide the service itself. There is no additional confirmation needed by the ISP from the customer to store this data.

-> Customer’s information

The information that is stored by default in Splynx software about a customer is as follows.

ISP administrators can create additional fields and store information about customers in these fields. In the case where these fields feature sensitive data, the ISP company should gain permission from the customer to use this data, Splynx is not responsible for gaining this permission, this is the total responsibility of ISP company.

-> Communication

Splynx software automatically sends emails and SMS to ISP customers in certain situations –  when an invoice is created, when payment is received or when an invoice is overdue and the service is put on hold. These emails are sent to end customers by the ISP because they are needed in order to provide the service and operate its business and as such they  are covered by the main service agreement. No additional confirmation from customers is needed in this instance.

Splynx does not provide a newsletter or similar service. If an ISP wishes to send promotion emails or SMS, they should obtain further confirmation from customers and this confirmation should be stored in the  newsletter/promotions platform.

Splynx stores all emails sent  to customers. Splynx also stores replies from customers in the Tickets section. Tickets can be displayed to customers or can be hidden and available only for administrators.

-> Passwords

Splynx stores all passwords in an SQL database in encrypted format. Any field that is defined as a password is automatically encrypted before saving to the database. The passwords are not displayed in logs or API calls. All customers’ passwords are by default hidden  to administrators. Customers can reset the passwords to their portals using the email reset form.

-> IP addresses

IP addresses and MAC addresses are important for an ISP company to provide access to the internet. IP addresses are stored in the Splynx database in section IPAM – IP address management. IP address and MAC address are unique and can be assigned to one customer at the same time.

-> Logs of sessions

Splynx stores information about IP sessions. The system stores a combination of IP address, the start session date time and the end session date time. This allows an ISP company to find out quickly which customers were using an IP address in a certain period of time.

Administrative access to Splynx software

-> Access to the system

Our software operates with two different access portals. The first is for administrative access and the second is for customer access.

Administrative access allows the administrators of an  ISP company to manage and work with customers’ information. Administrative access can be given by an  ISP to its own employees or to third-parties (for example resellers.) When a third-party receives access to Splynx software, administrative access can be limited and the third-party will only be able to view and work with customers that have a contractual relationship with the third-party.

The customer portal allows customer to login to an ISP system and view information about their services, invoices, payments, payment methods, statistics of internet usage and to  check the durations and cost of telephone calls

-> Access levels, roles and permissions

Access roles  – To achieve more granularity and tighten the security of user accounts, Splynx billing software supports access roles. The software makes it easy for an ISP administrator to assign permissions by providing pre-defined “roles” to choose from. These roles and their permissions are fully customizable at the administrator level.  By selecting one of the pre-defined roles (Super administrator Administrator, Customer creator, Financial manager, Manager, New Role) during user account creation, the billing system applies the relevant permission settings for that role.

Permissions – while the roles are predefined level of permissions, the Permissions themselves define which parts of the system can be displayed to or edited by administrator.

-> Logs of operations

Each single action of administrator in Splynx is tracked. View, Edit, Save, Delete, Rename, Change – anything an administrator does in the system is always saved in logs. All information is located in Administration -> Logs -> Operations. A search by customer, administrator or action is available in this section.

Technology description

-> SSL communication

Our platform is web-based. Administrators and customers access it via web browsers. Preferred communication is HTTPS. Our team helps to setup the SSL communication on ISP

-> Two factor authentication

Splynx software supports two-factor authentication. This is a configuration security feature that enables system administrators to implement customized access to Splinx software tools by setting Google authenticator or similar software.

-> Encryption of passwords

The passwords of administrators, customers, pppoe logins, routers etc are all  encrypted in the SQL database with the server’s key. Passwords are not sent in plaintext in any type of client-server communication (HTTP, HTTPS, API, Exports)

-> Remote backups

Splynx s.r.o. provides a remote backup service to its clients. This means that a client’s server connects to our cloud storage once every 6 hours and pushes the changes from database and application. All data from our clients and their customers is transferred over an OVPN encrypted tunnel. Data is transferred via Burb protocol which is also encrypted. The data is stored in the server in encrypted format. You can read more about our backup platform here – http://burp.grke.org/

Splynx support access

-> Administrative access

We have our own proprietary software solution that is used by our company for license management, backup management and which allows our employees to access client’s servers, provide assistance, resolve issues and restore data in the case of failure.

When the system is installed, our clients can request remote support. Splynx has a tool – splynx-remote-support – and when this tool is installed, the client’s Splynx server establishes an OVPN encrypted tunnel to our cloud platform. This allows us to access the client’s server on a private IP address.

When our employees access the web administration of a client’s Splynx server, they use the  administrator’s name, “splynx-remote-support”. They also use the password that is generated once every day. To access the client’s server, our administrator must also enter a one time code that is created every 30 seconds. This strict authentication allows us to ensure that in the situation where one of  our employees leaves the company and is therefore removed from the authentication system, they will not be able to access a client’s servers.

-> SSH remote access

Similar to administrative access is our SSH access. We connect to a client’s servers via a private IP address of the OVPN tunnel. Our support people connect to the client’s servers with Public Keys and no password. The Key is created based on the private key of the authentication server once every day. The sudo password is also changed once per day.

-> Splynx internal sub-system logs

All access by our employees to a client’s servers is logged. We store information about web access and SSH access. In addition, we also  store the history of commands that are run on the server and web actions are stored in each Splynx server under “Administration -> Logs -> Operations”

Huawei GPON configuration

We got Huawei MA5800-X7 device to help our customers with GPON integration. In this article you can find useful commands that help during configuration. Huawei instead of “show” uses “display” command. Also there is enable mode, similar to Cisco devices. If we need to delete part of configuration – use “undo some_command” instead of “no some_command”.

Let’s display basic information of the system :
display version
shows version of equipment and OS running on it.

this shows interfaces, for example MEth0, it’s now showing GPON ports.
display interface

The most important commands showing running and startup configurations.
display saved-configuration
display saved-configuration

get information about configured IPs and also routing table.
display ip interfaces
display ip routing-table

Let’s set user and config default route quickly :
ip route-static 0.0.0.0 0.0.0.0 10.0.0.1
ssh user "test" authentication-type password

Before configuring GPON and VLANs we need to install physical cards to chassis slots and initiate them with commands similar to :

This command is adding a card on board 0, slot 5. And it’s defining that it’s 16 port GPON card.
In our case we have one H901GPHF(GPON 16 port) and H901MPLA(SFP+ Ethernet) cards.
board add 0/5 H901GPHF

GPON configuration step by step :

1. SET VLANS. ID depends on design, let’s assume that 100 is management, 200 Internet, 300 voice and 400 is IPTV.
(config)
vlan 100 smart
vlan 200 smart
vlan 300 smart
vlan 400 smart

2. SET VLANs to UPLINK PORT.
Our UPLINK port is located in board 0, slot 0, SFP port 0. It means “0/8 0”

(config)

port vlan 100 0/8 0
port vlan 200 0/8 0
port vlan 300 0/8 0
port vlan 400 0/8 0

3. Defining DBA profiles. This defines upload speed (capability of the service on one ONT).
Here it says that upload for MGNT is maximum 1Mbps, for Internet is from 10Mbps to 200Mbps, Voice is 4Mbps and TV is up to 5 Mbps.

dba-profile add profile-id 100 profile-name "MGNT" type1 fix 1024
dba-profile add profile-id 102 profile-name "INTERNET" type3 assure 10000 max 200000
dba-profile add profile-id 103 profile-name "VOICE" type1 fix 4096
dba-profile add profile-id 104 profile-name "CUSTOM_TV" type1 fix 5192

4. Traffic tables configuration. Now let’s define real speed limitation.
We define MGNT to 1 Mbps Up and down, Voice to 4 Mbps, TV to 10 Mbps and Internet to 100/20 Mbps.

traffic table ip index 100 name "MGNT" cir 512 cbs 329680 pir 1024 pbs 329680 color-mode color-blind priority 7 priority-policy local-setting
traffic table ip index 101 name "INTERNET_100_DOWN" cir 10000 cbs 329680 pir 100000 pbs 329680 priority 0 priority-policy local-Setting
traffic table ip index 102 name "INTERNET_20_UP" cir 4096 cbs 329680 pir 20000 pbs 329680 priority 0 priority-policy local-Setting
traffic table ip index 103 name "VOICE_SMART" cir 2048 cbs 329680 pir 4096 pbs 329680 color-mode color-blind priority 7 priority-policy local-setting
traffic table ip index 104 name "CUSTOM_TV_COOL" cir 5192 cbs 329680 pir 10240 pbs 329680 color-mode color-blind priority 7 priority-policy local-setting

5. SYNC PROFILES/BUNDLE
(Config)
ont-lineprofile gpon profile-id 20 profile-name "FTTH-100"
Here we defined ont-lineprofile for FTTH-100 tariff bundle, that includes voice, IPTV, internet and management.

Following configuration lines are put under ont-lineprofile config :


tcont 1 dba-profile-id 100
tcont 2 dba-profile-id 102
tcont 3 dba-profile-id 103
tcont 4 dba-profile-id 104

gem add 1 eth tcont 1
gem add 2 eth tcont 2
gem add 3 eth tcont 3
gem add 4 eth tcont 4

gem mapping 1 1 vlan 100
gem mapping 2 2 vlan 200
gem mapping 3 3 vlan 300
gem mapping 4 4 vlan 400

commit

6. SERVICE PROFILE :

ont-srvprofile gpon profile-id 20 profile-name "FTTH-100"
ont-port pots adaptive eth adaptive
commit

7. ONT CONNECTION

Auto find can be enabled this way:

interface gpon 0/5
port 0 ont-auto-find enable

SET PROFILES to CUSTOMER :

interface gpon 0/5
ont add 0 1 sn-auth "4857544352E92103" omci ont-lineprofile-id 20 ont-srvprofile-id 20 desc "CUSTOMER-1"

8. The last are the most difficult commands – it’s defining service ports for each VLAN for ONT

service-port 1001 vlan 100 gpon 0/5/0 ont 1 gemport 1 multi-service user-vlan 100 tag-transform translate inbound traffic-table index 10 outbound traffic-table index 10
service-port 1002 vlan 200 gpon 0/5/0 ont 1 gemport 2 multi-service user-vlan 200 tag-transform translate inbound traffic-table index 102 outbound traffic-table index 101
service-port 1003 vlan 300 gpon 0/5/0 ont 1 gemport 3 multi-service user-vlan 300 tag-transform translate inbound traffic-table index 103 outbound traffic-table index 103
service-port 1004 vlan 400 gpon 0/5/0 ont 1 gemport 4 multi-service user-vlan 400 tag-transform translate inbound traffic-table index 104 outbound traffic-table index 104

When everything is configured, there are few helpful commands for displaying status :

Show all configured ONT devices
display ont info 0 5 0 all
display ont info summary 0/5
display service-port all

Show auto discovered ONTs
display ont autofind all

Show physical module installed on Huawei OLT chassis
display board 0

Show specific module installed
display board 0/SlotID
display board 0/5

Show all configuration
display display current-configuration

Show GPON dba profile
display dba-profile all

Show traffic table that is used for speed limitations
display traffic table ip from-index 0

Show service port
display service-port all

Show ont lineprofile
display ont-lineprofile gpon all

Show ont service profile
display ont-srvprofile gpon all

How to remove ONT from configured Huawei ONL.
First step is to delete all associated service ports:
undo service-port 1001
undo service-port 2001
undo service-port 3001
undo service-port 4001

Second step is to delete ONT itself :
interface gpon 0/5
ont delete 0 0


Should you have any questions regarding the GPON configuration or further information is needed, please contact us or schedule a call with our engineer.

 

Starting from Splynx v.3.0 we have added native support of IPv6. Find out more.

ISP Inventory management

Inventory management

Splynx brings Internet providers efficient and simple way to manage their inventory and stock. Feature is available from 2.0 release.
Inventory dashboard shows amount of items, products, suppliers and invoices in the system. Splynx allows to scan and quickly work with barcodes.

Suppliers are companies that sell equipment and issue invoices. Splynx stores and links items/equipment units with particular supplier’s invoice.

Products are routers, switches, cable units, antennas and similar. Administrator can define default price that will be used by system when equipment is rented or sold to customer. Vendors section store information about producers of equipment such as Mikrotik, Cisco, Radwin or Ubiquiti.

Items is the most used section, it’s a common table with all equipment units. Administrator is able to assign equipment units to other administrator, set as internally used, move units from stock to customer or mark them as broken and send back to supplier.

 

 

Barcodes. You can use any barcode reader and scan barcodes of equipment when entering it to stock. Then find simply equipment unit using it’s barcode. Our search barcode feature allows to select multiple barcodes and work quickly with barcode scanner instead of typing long useless numbers.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Should you have any questions or further information is needed, please contact us or schedule a call with our engineer.

Please check inventory management in video tutorial :