Building OpenStack Kolla Images from Source

There are several ways to deploy images for kolla. You can use docker hub, you can deploy them from a local private registry and you can build them as binary (rpm/packages) or from a combination known as source.

#Source Build

#OpenStack Basics
kolla-build --registry your.dockerrepo.com:4000 --push -t source fluentd kolla-toolbox cron chrony memcached mariadb rabbitmq dnsmasq keepalived haproxy -T 16 --tag train

# Projects
kolla-build --registry your.dockerrepo.com:4000 --push -t source  nova keystone cinder tgtd iscsid glance neutron openvswitch masakari placement aodh ironic horizon octavia manilla heat watcher  -T 16 --tag train

Having issues with kolla-build –push, so after all images are build I push them to my private registry.

#probably some docker specific commands for this but works.
docker images |grep your.dockerrepo.com| awk {'print $1'} | xargs -I {} docker push {}:train

Network Switch PC

Several years ago I was given a small network switch from my high school.  The switch was defective and dropped packets constantly so I wanted to give it new life.

Render

Removed the audio port riser from the motherboard so it would fit in the unit all the way.

Got a 1U heatsink for a server from dynatron.  Clears the lid just perfectly with back plate!

Fired up the system to do a load test to see if the blower was adequately powerful cooling.

Got our Intel 1G nic mounted with flexible PCI-e x4 adapter cable.

 

Soldered the internal terminals of the uplink port to a cat5 cable and plugged it in to our nic.

 

 

Made some lexan disk mounts and airflow containment.

Stacked sata cables are a challenge to figure out and get 2 compatible cables.

 

Soldered USB headers onto 4 of the ethernet ports and made ethernet to usb adapter cables!

Installing linux with software raid 1.

All done, you would never know!

Build the NAS from hell from an old nimble CS460

About 5 years ago we bought some nimble storage arrays for customer services… well those things are out of production and since they have the street value of 3 pennys I figured it was time to reverse engineer and use them for other purposes.

The enclosure is made by supermicro, its a bridge bay server which has 2 E5600 based systems attached to one side of the SAS backplane and 2 internal 10G interfaces. It appears they have a USB drive to boot an image of the OS and then they store configuration on a shared LVM or some sort of cluster filesystem on the drives themselves. Each controller has what looks like a 1GB NVRam to Flash pci-e card that is used to ack writes as they come in, and get mirrored internally over the 10G interfaces.

I plan to use one controller (server) as my Plex Media box and the other one for virtual machines. The plan right now is to use BTRFS for the drives and use BCache for SSD acceleration of the block devices. I can run iSCSI over the internal interface to provide storage to the 2nd controller as VM host.

To be continued.

— Update

Found out both of my controllers had bad motherboards, one was fine with a single cpu and would randomly restart, the other wouldn’t post. I feel bad for anyone still running a nimble, its a ticking time bomb. So I grabbed 2 controllers off ebay for $100 shipped, they got here today and both were good. I went ahead and flashed the firmware to the supermicro vanilla so I could get access to the bios. I had to use the internal USB port as nimbles firmware disables the rest of the USB boot devices and the bios password is set even with defaults so you can’t login. I tried the available password on the ole interwebs but nothing seemed to work, it only accepts 6 chars but the online passwords are 8-12.

 

Looks like bcacheFS is gonna be the next badass filesystem now that btrfs has been dropped by redhat. Will have full write offloading and cache support like ZFS so we can use the NVRam card. Speaking of write cache, I have an email into NetList to try and get the kernel module for their 1G NVram write cache card. Worse case scenario I have to pull it out of the kernel nimble was using…

As of writing this I have both controllers running CentOS7 installed to their own partitions on the first drive in the array, and I have /boot and the boot loader installed to the 4G USB drives that nimble had their bootloader installed to.

 

sda 8:0 0 558.9G 0 disk
sdb 8:16 0 558.9G 0 disk
sdc 8:32 0 558.9G 0 disk
sdd 8:48 0 558.9G 0 disk
sde 8:64 0 1.8T 0 disk
sdf 8:80 0 1.8T 0 disk
sdg 8:96 0 1.8T 0 disk
sdh 8:112 0 1.8T 0 disk
sdi 8:128 0 1.8T 0 disk
sdj 8:144 0 1.8T 0 disk
sdk 8:160 0 1.8T 0 disk
sdl 8:176 0 1.8T 0 disk
sdm 8:192 0 1.8T 0 disk
sdn 8:208 0 1.8T 0 disk
sdo 8:224 0 1.8T 0 disk
sdp 8:240 0 1.8T 0 disk
sdq 65:0 0 3.8G 0 disk

And I went ahead and created an MDRaid array on 6 of the spindle disk with LVM to get started messing with it. I need to get bcachefs compiled to the kernel and give that a go, will come with time!

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdj[6] sdi[4] sdh[3] sdg[2] sdf[1] sde[0]
      9766912000 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/5] [UUUUU_]
      [=>...................]  recovery =  7.7% (151757824/1953382400) finish=596.9min speed=50304K/sec
      bitmap: 5/15 pages [20KB], 65536KB chunk

Maybe I’ll dabble with iSCSI tomorrow.

— Update

Installed Plex Tonight, spent some time getting sonarr and other msc tools for acquring metadata and video from the interballs. Also started investigating bcache and bacachefs deployment in CentOS. http://10sa.com/sql_stories/?p=1052

Also started investigating some water blocks to potentially use water cooling on my NAS… its too loud and buying different heatsinks doesn’t seem very practical when a water block is $15 on ebay

 

–Update

I am def going to use water cooling, the 40mm fans are really annoying and this system has rather powerful E5645 cpus which have decent thermal output.   I found some 120MM aluminum radiators in ebay for almost nothing, so 2 blocks + fittings + hose is going to be around $80 per system.  I need to find a cheap pump option but I think I know what I’m doing there.

Heres a picture of one of the controller modules with the fans and a cpu removed.

 

A 80mm fan fits perfectly and 2 of the 3 bolt holes even line up to mount it in the rear of the chassis.  I will most likely order some better fans from delta with PWM/Speed capability so that the SM smart bios can properly speed them up and down.   You can see that supermicro/nimble put 0 effort into airflow management in these systems.  They are using 1U heatsinks with no ducting at all so airflow is “best efforts” I would guess the front cpu probably runs 40-50C most of its life simply due to the fact airflow is only created by a fixed 40mm fan in front of it.

 

–Update

Welp I got the news I figured I would about the NV1 card from NetList,  it is EOL and they stopped driver r development for it.  They were nice enough to send me ALL of the documentation and kernel module though, it supports up to kernel 2.6.38 so you could run latest centos 6 and get it supported.. maybe ill mess with that?  I attached it here incase anyone wants the firmware or linux kernel module driver for the Netlist NV1.  Netlist-1.4-6 Release

Free up disk space from deleted files under running processes.

A lot of the time a large log file will grow and need removed,  most the time these files cannot actually be “deleted” or “cleared” until the service releases its file descriptor.

 

Identify the file:

List files recently deleted that have not been released.

 

root@osc-1015 #> lsof -a +L1
COMMAND      PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK   NODE NAME
systemd-j   1059 root  txt    REG  253,4     278808     0  11628 /usr/lib/systemd/systemd-journald;570ba957 (deleted)
systemd-l   1451 root  txt    REG  253,4     584560     0  33117 /usr/lib/systemd/systemd-logind;570ba957 (deleted)
monitor     1617 root    5w   REG  253,4        500     0 261586 /var/log/openvswitch/ovsdb-server.log-20160404 (deleted)
monitor     1617 root    7u   REG  253,4        141     0     17 /tmp/tmpfsSX0WX (deleted)
ovsdb-ser   1619 root    7u   REG  253,4        141     0     17 /tmp/tmpfsSX0WX (deleted)
monitor     1722 root    3w   REG  253,4     474455     0 261589 /var/log/openvswitch/ovs-vswitchd.log-20160404 (deleted)
ceph-osd   20462 root  txt    REG  253,4   11589728     0  33573 /usr/bin/ceph-osd;570b9a69 (deleted)
ceph-osd   20686 root  txt    REG  253,4   11589728     0  33573 /usr/bin/ceph-osd;570b9a69 (deleted)
qemu-kvm  107850 qemu    8w   REG  253,4 2207794598     0 261623 /var/lib/nova/instances/8921a9ef-81c4-4a06-be00-7cad86bd6a1c/console.log (deleted)

in this instance I need to clear the console.log file

 

Release the kernel lock:

Now we will release its lock in the kernel.  The key parts here are the PID and FD,   We remove the write flag from the FD and use its ID.

 root@osc-1015#> : > "/proc/107850/fd/8"

 

Once ran the file is released and can be relocked by the process if it begins writing again.