jperrin.org

Thoughts on CentOS Stream

2020-12-08T00:00:00-08:00

I’m excited to see the CentOS project and Red Hat work together and collaborate around CentOS Stream, and I’d like to explain why I think this is a good move. I’ve been a member of the CentOS project for the last 16 years, and in that time I’ve had countless conversations with developers who were targeting enterprise deployments, but who wanted to push things just a little beyond what was currently available. In my early days with the project it was “I just need this new feature from PHP” or “I need this one option enabled in postfix”. This was so common it spawned an entire cottage industry of 3rd party repositories like Elrepo, IUS, nux-desktop, and even our own CentOSPlus. Often these were features Red Hat would include in a future version of RHEL, but the timing and communication around these features was a mystery. Red Hat never announced release dates or upcoming features and for many developers, even for those internal to Red Hat this was a PROBLEM.

When I joined Red Hat to work on CentOS full time, they outlined the goal pretty clearly: “We want to showcase our upstream community work we intend to put in our layered products”. Red Hat’s developers were using CentOS to do their upstream development work, and our role was to help them. This quickly became a problem, because the way to get new work into RHEL was Fedora, but that’s often not practical for a variety of reasons (Software Collections, modularity structure, release cadence, etc). Nothing here is new. Red Hat’s Josh Boyer and Brendan Conoboy spoke at length about this challenge in their Penrose Triangle talk at Flock in 2018

CentOS Stream represents several positive steps for Red Hat here.

It makes RHEL development more transparent and reliable.
It provides a way for ISVs and developers to contribute fixes and features.
It provides a way for the community to provide feedback.

Did you download the RHEL8 public beta? Did you notice that the python setup looked VERY different between the GA and the beta? If you spent the 6 months between the RHEL beta and GA developing your app to work, it’s possible you had the rug yanked out from under you. CentOS Stream solves that by providing constant updates so you can see what changes are coming, and adjust accordingly. Because RHEL’s development is now transparent to the public, devs shouldn’t be surprised by changes, they’ll finally be able to see them coming.

Seeing what Red Hat is doing is one thing, but for those users I mentioned earlier who just needed that one new feature - they now have a way to collaborate with Red Hat to make it a reality. CentOS Stream provides a way for users to submit pull requests and to make their case for why it should be included. This obviously doesn’t mean everyone will get their way, but it’s a stark improvement from the past.

Four years later with CentOS and Red Hat

2017-10-11T00:00:00-07:00

After 4 years of being at Red Hat, I still occasionally get questions that show not everyone understands what Red Hat means to CentOS, or what CentOS provides to Red Hat. They tend to think in terms of competition, like there’s an either or choice. Reality just doesn’t bear that out.

First and foremost, CentOS is about integration, and its important to know who the community is. We’re your sysadmins and operations teams. We’re your SREs, the OPS in your devops. We’re a force multiplier to developers, the angry voice that says “stop disabling SELinux” and “show me your unit tests”. We’re the community voice encouraging you to do things the right way, rather than taking an easy shortcut we know from experience will come back to bite you.

What we’re not is developers. We may pull in kernel patches, but we’re not kernel developers. We can help you do the root cause analysis to figure out why your app is suddenly not performing, but we aren’t the ones to write the code to fix it. We don’t determine priority for what does or doesn’t get fixed, that’s what Red Hat does.

The core distribution of CentOS is and has always been based on code written by Red Hat. This doesn’t mean it’s a choice of “either CentOS or RHEL,” because we’re in this together. CentOS provides Red Hat a community platform for building and testing things like OpenStack with RDO. We build new ecosystems around ARM servers. We provide a base layer for others to innovate around emerging technologies like NFV. But none of this would be possible without the work of RH’s engineering teams.

The community can build, organize and deliver tools in any number of creative ways, but ultimately the code behind them is being developed by engineers paid to address the needs of Red Hat’s customers. You can bet that RH is keeping an eye on what the CentOS community is using and building, but that doesn’t necessarily translate to business need.

We’re here to empower operators who want to experiment on top of the enterprise base lifespan. We’re here to bring tools and technology to those for whom it may be otherwise be out of reach. We’re here to take use cases and lessons learned from the community back to Red Hat as advocates. We’re happy to serve both audiences in this capacity, but let’s not forget how we buy the ‘free as in beer’.

Boosting CentOS server performance

2017-09-20T00:00:00-07:00

Last week I spent entirely too much time trying to track down a performance issue for the AArch64/ARM64 build of CentOS. While we don’t and won’t do performance comparisons or optimizations, this was fully in the realm of “something’s wrong here”. After a bit of digging, this issued turns out to impact just about everyone running CentOS on their servers who isn’t doing custom performance tuning.

The fix

I know most people who found this don’t care about the details, so we’ll get right to the good stuff. Check your active tuned profile. If your output looks like the example below, you probably want to change it.

[root@centos ~]# tuned-adm active
Current active profile: balanced

The ‘balanced’ profile means the CPU governor is set to powersave, which won’t do your server any favors. You can validate this by running cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor. To fix it, run the command below:

[root@centos ~]# tuned-adm profile throughput-performance

That’s it. This changes the governor to performance which should give you a pretty decent performance bump without any additional changes, and across all hardware platforms.If you’re interested in figuring out why the default setting is set this way, I’ll explain.

Why the default is “wrong”

The tuned package is installed and enabled by default. When it runs for the first time, it tries to automatically select the best performance profile for the system by running a couple of comparisons. It does this by checking virt-what output, and using the contents of /etc/system-release-cpe. The tuned file /usr/lib/tuned/recommend.conf is then used as the rulebook to see what matches and what doesn’t.

This starts to unravel a bit with CentOS, because the packages are derived from RHEL(Red Hat Enterprise Linux), and while RHEL may differentiate between server, workstation, etc CentOS does not. If you look carefully at the recommends.conf check for the throughput-performance profile, you’ll see that they check to see if the strings computenode or server exist in /etc/system-release-cpe. On CentOS, neither one does, because the distribution doesn’t make that distinction. Because these strings aren’t found, the fallback option of balanced is chosen.

Keeping an eye on CentOS performance with Grafana

2017-08-04T00:00:00-07:00

I’ve spent a bit of time setting up CentOS as a home router due to a number of frustrations with existing home routers on the market. This was both a good exercise and a bit of nostalgia from my early days with Linux. Once I’d finished getting the basics set up, I wanted a way to track various statistics. Network traffic, disk usage, etc. The venerable cacti is certainly an option, but that’s feeling a bit legacy these days. I’d prefer to use a newer tool with a more modern feel. This is what led me to Grafana. Below is a basic walkthrough for how I’ve set things up. This is a very basic install, that incorporates Collectd, influxdb, and Grafana all on the same host.

Collectd

What, you thought I’d jump straight into Grafana? We have to have data to collect first, and the best way to do that on CentOS is via collectd

The simplest way to get collectd on CentOS is via the EPEL repository. If you’re new to CentOS, or aren’t familiar with the Fedora’s EPEL repo, the command below is all you need to get started.

yum install epel-release

Now that the EPEL repo is enabled, it’s easy enough to install collectd in the same manner:

yum install collectd

There are a number of additional collectd plugins available in EPEL, but for our purposes here the base is enough. I would encourage you to explore the available plugins if your needs aren’t met by the base plugin.

Now that it’s installed, we need to configure collectd to send data out. Collectd generates the stats, but we need to put it someplace that Grafana can use.

In /etc/collectd.conf there are a few things we need to configure. In the Global section, uncomment the lines for Hostname, BaseDir, PIDFile, PluginDir, and TypesDB. You’ll need to modify Hostname, but the rest should be fine as the defaults. It should look something like the snippet below:

Hostname    "YourHostNameHere"
#FQDNLookup   true
BaseDir     "/var/lib/collectd"
PIDFile     "/var/run/collectd.pid"
PluginDir   "/usr/lib64/collectd"
TypesDB     "/usr/share/collectd/types.db"

Now that we have the basic app information set, we need to enable the plugins we wish to use. For my instance, I have syslog, cpu, disk, interface, load, memory, and network uncommented. Of these, the default values are fine for everything except network. The network plugin is used to send data to our collector, which in this case is influxdb. The network plugin will need to point to your influxdb server. Since we’re doing everything locally in this example, we’re pointing to localhost. It should look like the following:

<Plugin network>
  Server "127.0.0.1" "8096"
</Plugin>

InfluxDB

Now that we’re done with Collectd, we have to configure influxdb to pull in the data collectd is generating. Since influxdb isn’t in EPEL, we’ll have to pull this in from their repository. The command below makes it easy.

cat <<EOF > /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/centos/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF

Once that’s done, install the package with yum install influxdb and then it’s ready to configure. There are only a few things that need to happen in the /etc/influxdb/influxdb.conf config file.

In the [http] section of your /etc/influxdb/influxdb.conf, set enabled = true, and bind-address = ":8096". It should look like this:

[http]
  # Determines whether HTTP endpoint is enabled.
enabled = true

  # The bind address used by the HTTP service.
bind-address = ":8086"

Then scroll down to the [[collectd]] section and configure it like the section below:

[[collectd]]
  enabled = true
  bind-address = ":8096"
  database = "collectd"
  typesdb = "/usr/share/collectd"

At this point we can go ahead and start both services to ensure that they’re working properly. To begin, we’ll enable collectd, and ensure that it’s sending data. As with other services, we’ll use systemd for this. In the sample below, you’ll see the commands used, and the output of a running collectd daemon.

[jperrin@monitor ~]$ sudo systemctl enable collectd
[jperrin@monitor ~]$ sudo systemctl start collectd
[jperrin@monitor ~]$ sudo systemctl status collectd
● collectd.service - Collectd statistics daemon
   Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2017-08-02 11:22:18 PDT; 6min ago
     Docs: man:collectd(1)
           man:collectd.conf(5)
 Main PID: 18366 (collectd)
   CGroup: /system.slice/collectd.service
           └─18366 /usr/sbin/collectd

Aug 2 11:22:18 monitor collectd[18366]: plugin_load: plugin "disk" successfully loaded.
Aug 2 11:22:18 monitor collectd[18366]: plugin_load: plugin "interface" successfully loaded.
Aug 2 11:22:18 monitor collectd[18366]: plugin_load: plugin "load" successfully loaded.
Aug 2 11:22:18 monitor collectd[18366]: plugin_load: plugin "memory" successfully loaded.
Aug 2 11:22:18 monitor collectd[18366]: plugin_load: plugin "network" successfully loaded.
Aug 2 11:22:18 monitor collectd[18366]: Systemd detected, trying to signal readyness.
Aug 2 11:22:18 monitor collectd[18366]: Initialization complete, entering read-loop.
Aug 2 11:22:18 monitor systemd[1]: Started Collectd statistics daemon.

Now that collectd is working, start up InfluxDB and make sure it’s gathering data from collectd.

[jperrin@monitor ~]$ sudo systemctl enable influxdb
[jperrin@monitor ~]$ sudo systemctl start influxdb
[jperrin@monitor ~]$ sudo systemctl status influxdb
● influxdb.service - InfluxDB is an open-source, distributed, time series database
   Loaded: loaded (/usr/lib/systemd/system/influxdb.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2017-07-29 18:28:20 PDT; 1 weeks 6 days ago
     Docs: https://docs.influxdata.com/influxdb/
 Main PID: 23459 (influxd)
   CGroup: /system.slice/influxdb.service
           └─23459 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

Aug 2 10:35:10 monitor influxd[23459]: [I] 2017-08-12T17:35:10Z SELECT mean(value) FROM collectd.autogen.cpu_value WHERE host =~ /^monitor$/ AND type_instance = 'interrupt' AND time > 417367h GR...) service=query
Aug 2 10:35:10 monitor influxd[23459]: [httpd] 172.20.1.40, 172.20.1.40,::1 - - [12/Aug/2017:10:35:10 -0700] "GET /query?db=collectd&epoch=ms&q=SELECT+mean%28%22value%22%29+FROM+%22load_shortte...ean%28%22value%
Aug 2 10:35:10 monitor influxd[23459]: [I] 2017-08-02T17:35:10Z SELECT mean(value) FROM collectd.autogen.cpu_value WHERE host =~ /^monitor$/ AND type_instance = 'nice' AND time > 417367h GROUP B...) service=query

As we can see in the output above, the service is working, and the data is being collected. From here, the only thing left to do is present it via Grafana.

Grafana

To install Grafana, we’ll create another repository as we did with InfluxDB. Unfortunately the Grafana folks don’t keep release versions separate in the repo, so this looks like we’re using an EL6 repo despite doing this work on EL7.

cat <<EOF > /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/6/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

Now that the repository is in place and enabled, we can install grafana the same as the rest, yum install grafana. Once this is done, we can start working on the configuration. For this tutorial, we’re just going to set an admin username and password, because we’re doing this for a tutorial, and a single user instance. I would absolutely encourage you to read the docs if you want to start doing a bit more with grafana.

To accomplish this reasonably basic configuration, simply uncomment the admin_user and admin_password lines in the [security] section of /etc/grafana/grafana.ini, and set your own values. In this instance I’m using admin/admin, because that’s what you do in examples, right?

[security]
# default admin user, created on startup
admin_user = admin

# default admin password, can be changed before first start of grafana,  or in profile settings
admin_password = admin

Now you can start grafana with systemctl start grafana-server, and configure it via the web interface. After you log in for the first time, you’ll be prompted to configure a few things including a data source, and a dashboard. Since we’re doing this all on the localhost, you’ll be able to cheat and use the data source settings in the screenshot. Don’t worry, we’re nearly there and there’s only a little left to do.

Once you have the datasource configured, you’ll be prompted to create your first dashboard. While you can certainly do this, it’s a little intimidating for a first run with grafana. One easy solution to this is to import one of the available templates offered on Grafana’s website. In my case, I opted to use the Host Overview. It provides a nice group of metrics and graphs as a base to use and build from.

Once you’ve gotten everything set up, it’s now down to personal preference and further tinkering. Once again I would very much recommend reading the documentation because there are a wealth of options and changes I didn’t touch on for this intro.

Hands-on with a Minnowboard Dual-e

2017-07-22T00:00:00-07:00

I recently got my hands on one of the dual ethernet Minnowboards from Adi Engineering. I’ve been on the hunt for a nice low power, small form factor development board for a while, but nearly everything available in my price range comes with a single network card.

This board is a bit of an improvement over previous Minnowboards, with an M.2 slot, and dual ethernet, but lacks the onboard emmc available on previous versions. Since I had a few spare m2 ssds around, it’s not a huge deal for my purposes. Once I’ve gotten through testing this board out, the plan is to build out a demo cluster to bring around to various conferences to showcase what we’re currently doing with the distribution, so you may hear a bit more from me on this in the future.

A Fresh Start

2017-07-21T00:00:00-07:00

For the last few years, I’ve not really cared at all about a semi-permanent slice of home on the internet. I’ve stuck mostly with twitter and only the occasional blog post, usually on someone else’s platform. A few folks like Ben Cotton have tried to reform me. They’ve gotten me to the point where I’m starting to feel a little guilty about being a digital vagrant…and so here we are.

I can’t promise miracles, but I am going to try to write more frequently, and rebuilding some proper website tooling seemed like an interesting way to go about preparing. This time, if I stop maintaining this little website slice, I’ll at least have the decency to feel guilty about it.