So for the people who don’t know what MIT Kerberos is… congratulations. You are a happy person. Thank you for the visit… but now… go away… I don’t want to be accused of damaging the brain of a fellow IT friend. I was a happy person before I knew Kerberos… #LEL.
So… in one of our customers I found out that the servers were configured with SSSD via Kerberos keytab. This was awesome because we already had the AD groups coming correctly for HUE via SSSD.
Well… actually… not that awesome. Because this creates a huge problem for the entire Cloudera cluster.
Why? Because both SSSD and Cloudera CDH systems read the /etc/krb5.conf… and if you want to create a local Kerberos server… you are basically… screwed.
The krb5.conf has a parameter called default_realm. For the local Kerberos server you need to set it up to your REALM. But then again… SSSD was doing the same… but with the AD Kerberos realm… do you see the problem now? Yeah… big one.
We found out that the guys from Cloudera forgot to set up a freaking variable in their system in order to read the krb5.conf from somewhere else… a different path from /etc/krb5.conf. This crap must be hardcoded somewhere. BAD CLOUDERA BAD!!!
So… if you need to use 2 different krb5.conf files with CDH, how do you do it? Now this is the part where you embrace the almighty god Thor, and trust me that this… this is for your own good!
Note: this was made for CDH 6.0.0.
Place your custom krb5.conf in the “jre/lib/security” subdirectory of your JDK’s directory. Example:/usr/java/jdk1.8.0_141-cloudera/jre/lib/security/krb5.conf
Make sure it is readable for everyone in terms of file permissions (so that all service users can read from it).
In every CDH service change “Environment Advanced Configuration Snippet” (in Cloudera Manager), add something like this:
KRB5_CONFIG=/usr/java/jdk1.8.0_141-cloudera/jre/lib/security/krb5.conf
For all agents in your cluster, add the following to /etc/default/cloudera-scm-agent and /etc/default/cloudera-scm-server:
export KRB5_CONFIG=/usr/java/jdk1.8.0_141-cloudera/jre/lib/security/krb5.conf
Restart EVERYTHIIIIING (cluster, management service, agents, Cloudera Manager) and pray that you have everything correct. If you don’t, go to the closest pharmacy and buy a pack of Xanax. Trust me… you’ll need it. Go buy it anyway. Thank me later.
Give it a quick check in all the main logs (agents, server, services) to see if something is screaming (it will…). If something is dead, check if you didn’t forget to set the Snippet variable somewhere, or in the agent.
Ok, so… CDH done, agents and server done. What’s next?
Next you hammer kerberos a little bit more. This time we strike with our mighty hammer the krb5kdc and kadmin services. On my case, the files were /etc/sysconfig/krb5kdc and /etc/sysconfig/kadmin. This will allow that when the services start, they will read the alternative krb5.conf file, instead of the /etc/krb5.conf by default.
Give it a check in the service file in order to see where it is in your system.
root@sl000056:~$ systemctl status krb5kdc
● krb5kdc.service - Kerberos 5 KDC
Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2018-10-24 12:14:18 WEST; 6 days ago
Main PID: 22321 (krb5kdc)
CGroup: /system.slice/krb5kdc.service
└─22321 /usr/sbin/krb5kdc -P /var/run/krb5kdc.pid
root@sl000056:~$ cat /usr/lib/systemd/system/krb5kdc.service [Unit] Description=Kerberos 5 KDC After=syslog.target network.target [Service] Type=forking PIDFile=/var/run/krb5kdc.pid EnvironmentFile=-/etc/sysconfig/krb5kdc ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS ExecReload=/bin/kill -HUP $MAINPID [Install] WantedBy=multi-user.target root@sl000056:~$ cat /etc/sysconfig/krb5kdc KRB5KDC_ARGS= KRB5_CONFIG=/usr/java/jdk1.8.0_141-cloudera/jre/lib/security/krb5.conf
And that’s “basically” it… restart kerberos, kadmin and pray! Don’t forget to take one or two xanaxs.
Next time that you want to interact with both services (for example to kadmin.local and add principals or whatever makes you high), remember to export these two in your session:
export KRB5_CONFIG=/usr/java/jdk1.8.0_141-cloudera/jre/lib/security/krb5.conf export KRB5CCNAME=/tmp/krb5cache
Have fun!
