Seamless Migration: Upgrading FreeIPA Data from Version 4.6.8 to 4.10.1 - Proactive Insights and Support For Open-Source Applications

Problem:

The problem is that commit logs in the production Cassandra cluster are accumulating excessively without being deleted, leading to a full filesystem and subsequent database crashes.

Process:

FreeIPA-Replication-Troubleshooting
We have tried to reproduce the setup by performing the following actions.

Installing FreeIPA Server on CentOS 7 server. Which had the default version of FreeIPA being 4.6.8 as seen below:

[root@centos7 ~]# ipa --version
  VERSION: 4.6.8, API_VERSION: 2.237

Installing FreeIPA Server on Rocky Linux 9, the default OS upstream version is 4.10.2

[root@rocky9 ~]# ipa --version
  VERSION: 4.10.2, API_VERSION: 2.252

Enrolling Rocky Linux 9 server as client to CentOS 7 FreeIPA installation – this was successful. For the customer the failure was at this point!.
On attempting to configure Rocky Linux 9 server as FreeIPA replica – this failed with the following error message:

[3/30]: creating ACIs for admin
  [4/30]: creating installation admin user
  [5/30]: configuring certificate server instance
  [6/30]: stopping certificate server instance to update CS.cfg
  [7/30]: backing up CS.cfg
  [8/30]: Add ipa-pki-wait-running
  [9/30]: secure AJP connector
  [10/30]: reindex attributes
  [11/30]: exporting Dogtag certificate store pin
  [12/30]: disabling nonces
  [13/30]: set up CRL publishing
  [14/30]: enable PKIX certificate path discovery and validation
  [15/30]: authorizing RA to modify profiles
  [16/30]: authorizing RA to manage lightweight CAs
  [17/30]: Ensure lightweight CAs container exists
  [18/30]: Ensuring backward compatibility
  [19/30]: destroying installation admin user
  [20/30]: starting certificate server instance
  [21/30]: Finalize replication settings
  [22/30]: configure certmonger for renewals
  [23/30]: Importing RA key
  Error storing key "keys/ra/ipaCert": CalledProcessError(Command
  ['/usr/libexec/ipa/custodia/ipa-custodia-ra-agent', '--import', '-'] returned
  non-zero exit status 1: 'Traceback (most recent call last):\n File
  "/usr/libexec/ipa/custodia/ipa-custodia-ra-agent", line 8, in \n
  main(ra_agent_parser())\n File "/usr/lib/python3.9/sitepackages/ipaserver/secrets/handlers/pemfile.py", line 117, in main\n
  common.main(parser, export_key, import_key)\n File "/usr/lib/python3.9/sitepackages/ipaserver/secrets/handlers/common.py", line 73, in main\n
  func(args, tmpdir, **kwargs)\n File "/usr/lib/python3.9/sitepackages/ipaserver/secrets/handlers/pemfile.py", line 72, in import_key\n
  ipautil.run(cmd, umask=0o027)\n File "/usr/lib/python3.9/sitepackages/ipapython/ipautil.py", line 599, in run\n raise
  CalledProcessError(\nipapython.ipautil.CalledProcessError:
  CalledProcessError(Command ['/usr/bin/openssl', 'pkcs12', '-in',
  '/tmp/tmpbvpa2106/import.p12', '-clcerts', '-nokeys', '-out',
  '/var/lib/ipa/ra-agent.pem', '-password', 'file:/tmp/tmpbvpa2106/passwd']
  returned non-zero exit status 1: 'Error outputting keys and
  certificates\\n802B96BA7E7F0000:error:0308010C:digital envelope
  routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global
  default library context, Algorithm (RC2-40-CBC : 0), Properties ()\\n\')\n')
  [error] FileNotFoundError: [Errno 2] No such file or directory:
  '/var/lib/ipa/ra-agent.key'
  Your system may be partly configured.
  Run /usr/sbin/ipa-server-install --uninstall to clean up.
  [Errno 2] No such file or directory: '/var/lib/ipa/ra-agent.key'
  The ipa-replica-install command failed. See /var/log/ipareplica-install.log for
  more information
  [root@ipa02 ~]# ls /var/lib/ipa/ra-agent.key
  ls: cannot access '/var/lib/ipa/ra-agent.key': No such file or directory

Solution:

Upon further diagnosis, we noted that CentOS 7 FreeIPA version to RHEL 9 or Rocky Linux 9 FreeIPA replication is not possible.
The solution that we tested to be working in switching from CentOS 7 to Rocky Linux 9 FreeIPA
upgrade is as follows:

First use Rocky Linux 8 servers – Enroll as client and promote to replica(s); version 4.6.8
to version 4.9.12
Later switch off CentOS 7 servers – decommission once everything is tested.
Then redo the steps to switch from Rocky Linux 8 to Rocky Linux 9 (this will be version
4.9.12 to version 4.10.2 )

Even if the customer was to attempt CentOS 7 to Rocky Linux 9 process, it will still fail at
replication stage. That’s why we recommend CentOS 7 –> Rocky Linux 8. Then maybe later
Rocky Linux 8 –> Rocky Linux 9.

Adding Rocky Linux 8 machine as FreeIPA Replica
Below are the steps that were used to add Rocky Linux 8 machine as a replica to (CentOS 7
powered FreeIPA domain).

Requirements:

Both CentOS 7 and Rocky Linux 8 machines must be on the same domain (based DNS
domain) – e.g the one used in this documentation is lab.example.com
The assumption here is for the benefit of simplicity is that;
CentOS 7 hostname is: centos7.lab.example.com
Rocky Linux 8 hostname is rocky8.lab.example.com
Where: lab.example.com is FreeIPA base domain / Domain name. This should be substituted
accordingly.
Rocky Linux 8 machine should have FQDN set, DNS server configured to CentOS 7 FreeIPA
server

Step 1: Enroll the server as client

Set correct hostname (within FreeIPA base domain)
[root@rocky8 ~]# hostnamectl set-hostname rocky8.lab.example.com
Update DNS server record inside the /etc/resolv.conf file
# Just an example
[root@rocky8 ~]# vim /etc/resolv.conf
search lab.example.com
nameserver CentOS7_FreeIPA_IP
Nmcli command can be used to set statically.
[root@rocky8 ~]# nmcli connection show
NAME UUID TYPE DEVICE
enp1s0 bd009451-539e-41bc-852f-d42ffe33d91c ethernet enp1s0
[root@rocky8 ~]# nmcli connection modify ipv4.dns
Install FreeIPA Server and Client packages.
[root@rocky8 ~]# dnf module reset idm
[root@rocky8 ~]# dnf -y install @idm:DL1
[root@rocky8 ~]# dnf -y install freeipa-server freeipa-server-dns freeipaclient
Confirm the version of FreeIPA installed.
[root@rocky8 ~]# ipa –version
VERSION: 4.9.12, API_VERSION: 2.251
Open the ports in the firewall
[root@rocky8 ~]# firewall-cmd –add-service={freeipa-ldap,freeipaldaps,dns,ntp,freeipa-replication} –permanent
[root@rocky8 ~]# firewall-cmd –reload
Enroll Rocky 8 server as client, don’t pass other options for autodiscovery
[root@rocky8 ~]# ipa-client-install

Step 2: Promote to FreeIPA Replica

Commands executed on CentOS 7 FreeIPA master
Login to CentOS 7 server (current master) and allow freeipa-replication service if the
firewalld is active.
[root@centos7 ~]# firewall-cmd –add-service=freeipa-replication –permanent
[root@centos7 ~]# firewall-cmd –reload
Get Kerberos ticket for command line management.
[root@centos7 ~]# kinit admin
[root@centos7 ~]# klist
FreeIPA server are placed in ipaservers host group as shown:
[root@centos7 ~]# ipa hostgroup-find ipaservers
Check the current list of servers which can act as domain controllers.
[root@centos7 ~]# ipa hostgroup-show ipaservers
We need to add our FreeIPA replication host (Rocky Linux 8) to the ipaservers host group.
[root@centos7 ~]# ipa hostgroup-add-member ipaservers –hosts
rocky8.lab.example.com
Confirm it’s added with the following command
[root@centos7 ~]# ipa hostgroup-show ipaservers
Commands executed on Rocky Linux 8 client (to become replica)
Ensure firewall rules are set correctly
[root@rocky8 ~]# firewall-cmd –add-service={freeipa-ldap,freeipaldaps,dns,ntp,freeipa-replication} –permanent
[root@rocky8 ~]# firewall-cmd –reload
Install FreeIPA server package and integrated DNS if this is needed.
[root@rocky8 ~]# dnf -y install freeipa-server freeipa-server-dns
Then setup the machine as FreeIPA server – the options provided can be adjusted
accordingly.
[root@rocky8 ~]# ipa-replica-install –setup-ca –setup-dns –auto-forwarders
The execution was successful as expected unlike the one for Rocky Linux 9 joining CentOS 7
realm.
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
Restarting named
Updating DNS system records
Global DNS configuration in LDAP server is empty
The client can use ‘dnsconfig-mod’ command to set global DNS options that would override settings in local named.conf files
Configuring SID generation
[1/7]: creating samba domain object
[2/7]: adding admin(group) SIDs
[3/7]: adding RID bases
[4/7]: updating Kerberos config
‘dns_lookup_kdc’ already set to ‘true’, nothing to do.
[5/7]: activating sidgen task
[6/7]: restarting Directory Server to take MS PAC and LDAP plugins changes
into account
[7/7]: adding fallback group
Done.
The ipa-replica-install command was successful
From the CentOS 7 system the replica could be seen as a member of the ipaservers group.
[root@centos7 ~]# ipa hostgroup-show ipaservers
Host-group: ipaservers
Description: IPA server hosts
Member hosts: centos7.lab.example.com, rocky8.lab.example.com

Conclusion:

We recommend enrolling Rocky Linux 8 systems into the FreeIPA domain before switching to Rocky Linux 9. If the steps shared are followed correctly while replacing used variables / parameters to suit the customer environment, the process was successful.