1. Initialize a thread to acquire data
   1. When data arrives, put it into a data structure
   2. Place the data structure on a queue
   3. Notify another thread to process queued data
2. Initialize a thread to process data
   1. Wait for data on the queue

Linux process threads can share data access but can’t, without help, guarantee data integrity.  This means that one thread may be in the process of updating data while another is reading it.  Typically data integrity of a shared data area is provided through the use of a mutex (mutual exclusion lock).  This is fine for many cases but the lock just stops a thread from accessing the data while another thread accesses it.  What this doesn’t handle is synchronization of data availability. 

For example, if thread 1 locks access to a link list to add a record and then releases the lock that doesn’t tell a separate thread how many links are there to be processed.  So the second thread spins trying to pull links from the list.  If the first thread has not put any links on the list then the second thread just spins again, very fast (unless you use some artificial delay).  This is wasteful because it’s not synchronized.  The second thread only wants to process the list if there is something on the list.  The first thread has to tell the second thread that there is work to be done.

There are clever ways to do this with just mutexs where you lock both the list and a counter.  But there is a simpler way of dealing with the problem.  A way that doesn’t even require checking the value of the counter directly: a semaphore. 

A semaphore is a communications mechanism between threads.  One thread updating the semaphore (effectively adding one to the counter) is mutually exclusive to another thread reading the semaphore.  Reading the semaphore is effectively like having the second thread asking is there any more data I need to handle?  If not, the second thread can block until there is data, which happens when the first thread posts to the semaphore. 

The semaphore is similar to polling on a file descriptor where the poll blocks until there is data to be read.  With a semaphore the blocking happens until the semaphore has a value greater than zero.  Breaking the block simply requires posting to the semaphore, which increases its value.

All of this is easier to show with code.  First, we want code for adding a link to a list and code for removing a link.  Think of the first as a thread reading data from the network.  It takes the inbound data and places it in a structure that it then adds to the link list.  This code locks access to the list and then adds data to it.

pthread_mutex_lock(queueMutex);

queueData(rawData);

pthread_mutex_unlock(queueMutex);

This code guarantees that adding the link will complete before another thread can check the link list.  The second thread can get the link in a similar manner.

while( 1 ) {

    pthread_mutex_lock(queueMutex);

    data_link  = popData();

    pthread_mutex_unlock(queueMutex);

}

The mutexes prevent the two threads from stepping on each other (and in reality the lock/unlock would actually happen in the popData() and queueData() functions but doing it as shown here doesn’t require writing the functions), a problem that gets bad when the link list has no members.  But the second thread still needs to know when to try and pop data from the list otherwise the second thread will spin very fast and eat up CPU cycles.  The first thread therefore needs to post a semaphore, which bumps the semaphore count by one.

pthread_mutex_lock(queueMutex);

queueData(rawData);

pthread_mutex_unlock(queueMutex);

sem_post(&dataSem);

Remember that a semaphore call is effectively wrapped by a mutex preventing two threads from accessing the semaphore at the same time.  The second thread now just needs to test the semaphore at the top of each loop.

while( 1 ) {

    sem_wait(&dataSem);

    pthread_mutex_lock(queueMutex);

    data_link  = popData();

    pthread_mutex_unlock(queueMutex);

}

Now the second thread blocks on the semaphore until has a value of least 1.  When the semaphore has at least one, the wait returns and decrements that semaphore by one.  Now there is synchronization between the two threads.  The first thread can queue up as many links as it wants and the second will drain that list as long as there is at least one link.  And the second thread doesn’t need to take up a lot of CPU cycles by spinning really fast when there is no data on the list.  It only runs when there IS some data on the list!

Now, about that while(1) statement:  This code is inside the function spawned by the thread creation.  So how do we exit the loop?  Well, when the code that wants to shutdown needs to kill off that thread it can set a variable and post to the semaphore. So now that loop would look more like this.

while( isActive() ) {

    sem_wait(&dataSem);

    if ( !isActive() )

        break;

    pthread_mutex_lock(queueMutex);

    data_link  = popData();

    pthread_mutex_unlock(queueMutex);

}

The test of isActive is placed inside a thread-safe function that wraps reading and writing of the variable with a mutex as well.  And the code that kills the thread just looks something like this.

setActive(FALSE);

sem_post(&dataSem);

You might wonder why the semaphore was used for the link list and for the thread loop.  Aren’t those unrelated?  Why yes, they are.  The semaphore doesn’t have any association with the link list.  It’s just a counter we’re using to keep track of adding and removing link entries.  But it’s a special counter because it’s thread-safe:  adding to it and taking from it is guaranteed to be safe from any thread.  And it can be used to block a read, one that wants to wait for some work to do.  The semaphore a communication mechanism between the thread requesting work to be done and the thread that needs to do the work. 

A more expansive example can be found in the piboxd daemon’s queueProcessor and msgQueue modules.  The latter is used to manage link lists of data while the latter is used to processes data queued on one of those lists.  The queueProcessor thread loop waits on a semaphore that is updated by the msgQueue functions when data is added to a list.

So you see, dealing with threads is not just about locking access to data with mutexes.  It’s also about communicating the changes to data via semaphores. 

The post Thread Management with Mutexes and Semaphores first appeared on Michael J. Hammel.

The problem with wanting to move is knowing you will have to sell your house.  That means keeping it in a perpetual state of staging, waiting to put it on the market as soon as a job offer arrives.  I interviewed all over, mostly targeting Seattle, Portland, Austin and Boulder.  Lots of interest.  Lots of interviews.  No offers.  I’m assuming (mostly, but not completely) because I was looking for relocation assistance.  This went on till late August when NetApp in Boulder made an offer.  Perfect timing as the severance package was starting to run a bit lean.  Time to get the house on the market.

Housing prices in Colorado are pretty high.  Selling your house normally means hiring a realtor to market it for you.  They get 3% of the sale price.  Toss in the buyers agent and that’s another 3% out of our sale price.  With really high housing prices that’s a really big chunk.  Too big.  So we started with a For-Sale-By-Owner.  There’s a web site for that.  And it’s not that hard to get started.  But it’s hard to market.  Turns out FSBO’s are not something realtors will look at much to bring you buyers.  And you don’t end up in the MLS (a monopoly if ever there was one) so realtor’s don’t even know about you.  Zillow will post it but marks it like it’s something to be wary of.  We got one bite.  And it was, as many are in military-based Colorado Springs, a VA loan. 

Ugh.  VA loans are difficult.  The buyer has to bring a teeny-tiny amount as down payment, meaning they usually don’t have anything extra.  VA inspections are tougher and any fixes come out of the seller’s pocket.  No negotiation there.  Screw it.  No VA loans.  And we dropped the FSBO.  Time for a flat-fee listing agent.

A flat-fee agent charges a flat fee to list you in MLS.  They don’t show your house.  But for the savings that’s something we could do easily enough.  Our agent did a great job, including providing a guiding hand through the whole sale process.   Colorado, technically, doesn’t allow a flat-fee service with limited services but she gets around that by basically doing everything a normal listing agent would do – for way less.  And so we sold our house.  That took another 2 months to complete.  In the mean time I’d started work in Boulder, was commuting up on Monday and back on Friday (lots of cheap hotel points accumulated) and helping the wife get the house ready.  Then came the move.

We hired a truck but no movers.  Our mistake.  We found someone at the last minute, who managed to find two more guys who hired out for day work.  They did a great job, but I worked 3 days straight (okay, a few hours sleep) filling the truckS (multiple, we needed a second one, we discovered as we’re loading the first).  We got out about 1/2 hour before the signing of the sale, exhausted to the point of barfing.  We drove both cars, dragging U-Hauls, with the dogs up to Superior, CO to a Residence Inn were we lived for a month trying to find a new home.   About 5 weeks later, after much negotiating, we closed on a new home in south west Broomfield.  Now we’re into late November, just before Thanksgiving.

We start unpacking and everything is going well.  We’re still exhausted but gotta get settled in as quick as possible:  my wife hasn’t been able to work since April!  Her business is run out of the house and she has no space set up to work.  Not to mention our daughter is coming to visit the first week of January. So we’re rushing to get everything done as the holidays fly towards us.  We’re moving along, my week off from work for the end of the year is one day away and then:  our Bailey, now the oldest surviving pup at 14, has a stroke.

The weird thing about dogs and strokes is that dogs recover from them much better than we do.  Bailey was nearly lifeless four days before Christmas.  Just her eyes moved.  We didn’t have a vet yet.  We scrambled to get her body into the back of the car, drove to one vet (my wife nixed) and then another.  That vet examined her and said, “Let’s give her a chance.”   He gave us meds, starting with steroids.   A week later she was about 90% back to normal, but with an incontinence problem.  It’s taken 3 more weeks, and trying a variety of drugs and hormones, to get that under control.  And now she’s walking around better than before she had the stroke.  We’re amazed, relieved, exhausted.  That first week after the stroke I had to dead-lift a 90lb dog four or fives times a day to take her to potty or to the vet.  On the bright side, I got in shape quick.

So here we are, the third week of January 2018 and all my wife and I can say about 2017 is:  good riddance!  2018 will be better, though we know Bailey is old and may not see another new year.  But she’s happy.  And so are we.  We’re out of Colorado Springs, into a dog friendly neighborhood with a new house and a great new job.  Life is good.

So now, back to that list of things to do in 2017….

The post Where’s Michael? first appeared on Michael J. Hammel.

In 2014 Google announced plans for pushing HTTPS migration for all web sites.  The biggest draw for web site owners is that Google bumps up your rankings if you’re using HTTPS.  There have been multiple extensions to this plan, from the EFF/Tor foundations HTTPS Everywhere Extension to EncryptAllThings.  Despite running multiple web domains, I’ve delayed joining the worldwide bandwagon jumping because I’m not that familiar with what is involved.  I’m neither an expert in HTTPS nor an expert in configuring web servers.  I also don’t care much about my rankings since I’m really not selling anything – I’m just sharing what I know.  But the gist of it is this:  if you run a web site, switch it from HTTP to HTTPS.  It’s just one letter in a URL.  Sounds simple enough, right?  Oh, how I wish it was.

Migration from HTTP to HTTPS takes a number of steps.  I just completed this task for all my domains, which run as virtual hosts under the Apache web server on my colocated server.  There isn’t a simple list of things to do anywhere.  There are lots of lists of things to do under specific situations.  My situation is CentOS and Apache.  Let me outline the steps here before I dive into the actual process.

1. Get and install a certificate
2. Enable SSL on your web server
3. Enable SSL for your domains
4. Migrate web sites
5. Redirect HTTP
6. Validate your work
7. Take a vacation

The last step you earn after completing the rest.  Anyone who denies you this last step must be eliminated.  You’ll understand why once you start down the twisty little configurations.

Get a Certificate

This article’s title suggests the main topic is certbot, the tool for acquiring and managing certificates, and the article introduction suggests using that tool is hard.  Well, it is and it isn’t.  It’s hard if you aren’t set up correctly and it’s hard to find out how to get set up correctly.  It’s not hard once you’re set up.  So let’s get set up.

HTTPS is the web protocol that wraps communications between a client, say a web browser, and a web server inside a nice layer of unintelligible gibberish that makes it harder for the bad hombres (no, not the ones Trump is trying to ship out, but actual bad guys)  to sniff around your messages and extract useful tidbits.  The protocol does this be running the stuff it would send through cryptographic algorithms to ensure privacy and integrity of the message while validating the authentication of the opposite end.  To do all this neat stuff it needs two things on the server-side:  an SSL certificate and a private key.

The private key is a secret.  Only your hairdresser knows for sure… wait, that’s different.  Only your server knows the secret.  If you let it out to anyone else then it’s not a secret and all the encryption in the world won’t save your silly buns.  So keep it secret, Frodo.  Keep it safe.

An SSL certificate contains authentication information along with a public key that matches with the private key.  It is used by the client side to establish trust with the server and allow encryption to be used that both sides known how to decrypt. 

To get these two magic files you need to talk with a Certificate Authority (CA).  A CA is someone who can sign a certificate, making it something the world can trust.  Who are these wizards of trust?  No idea.  But I know someone who does:  certbot.  This is a handy little tool for Linux users (if you’re  a Windows user and your reading my blog then you’ve lost your way).  It’s a command line utility that will handle all the nasty bits of getting a CA to sign a certificate stating your little nest of websites is a clean and decent place to visit.  Great.  So how do we use it?

Well, all the googling in the world will tell you certbot is an automated system that just needs to be run like so:

sudo ./certbot-auto

This will query you for necessary credentials, including the list of domains you want covered by a single certificate, make the request to get a signed certificate and install it on your server.  It will even update your Apache configuration files.  After this, you’re one of the good guys again, right?

Not exactly.  This command never worked out of the box for me.  There were a couple of gotchas I need to pass along.  First, make sure all your domains have proper DNS entries.  If you have one (as I did) that had goofy third or fourth level resolvers configured then this aint gonna work.  No how, no way.  Fix that first. 

Second, every web domain you have configured under Apache has a web root.  This is the directory where your HTML files (or php or javascript or whatever) reside.  If you try to run certbot-auto as described it fails because there is something missing from these web roots.  What might that be?  Google around – you won’t find it. But you can figure it out if you stop waiting for everything to be automatic and try something more manual, like this:

sudo certbot-auto certonly –webroot

This tells certbot that

1. We only want the certificate files installed in a directory somewhere
2. but don’t update our Apache configurations

What you get from this is an error message that looks like this:

Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.piboxproject.com (http-01): urn:acme:error:unauthorized :: The client lacks
sufficient authorization :: Invalid response from
http://www.piboxproject.com/.well-known/acme-challenge/UwffOaP-jo2Vs4NB-JceCq6UlspsD1KI3A6ZFveCuHA: "<!DOCTYPE HTML
PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p"

Buried in here is the missing directory:  .well-known/acme-challenge.  So you need this directory in every web root.  And it needs to be readable by the web server and writable by who ever is running certbot (in my case that was always the root user).  The certbot tool stuffs a file in there that the remote server – the one handling generation of your certificates – can look retrieve. 

So you create the missing directories and give them the proper permissions for every web root that will be associated with the certificate you’re requesting with certbot and you’ve fixed any DNS problems.  Now you’re set up.  So run the tool again. This time everything is hunky-dory and you escape certbot land none the worse for wear.   What you end up with is a directory called /etc/letsencrypt that contains a bunch of directories and some files.  Note that I said some files.  Not just two.  No, not just the key and the certificate.  No.  You get 14 files.  Or maybe 18.  I ran it multiple times so I’m not sure which.  And no explanation as to why so many files.  But we’ll come back to that later.

For now, you’re done with getting your certificate(s).

Enable SSL on your web server

Remember: I’ m not an Apache expert.  There may be (are likely) better ways of doing this.

Modern installations of Apache will have a configuration directory named /etc/httpd/conf.d.  Within this directory you will stuff your web domain configurations, one per domain with extras to redirect from variations on the main URL to the one you really want people using. 

Within this directory make sure you have a file called ssl.conf.  It’s what enables SSL (re: HTTPS) on your server in general.  But you need to update your virtual host configurations to actually use it.  You do that by changing your VirtualHost lines that look like this:

<VirtualHost www.my.domain>

or

<VirtualHost www.my.domain:80>

to

<VirtualHost www.my.domain:443>

But wait!  You don’t want to change your existing configurations.  Make copies of the Apache virtual host configurations for your new :443 configurations.  This is so you can support both HTTP and HTTPS while you complete the migration process. Also, because the HTTPS configurations will use a specific port you need to make sure your existing configurations include the :80 port.  You can’t mix configurations that have and don’t have port specifications. 

But don’t redirect from HTTP to HTTPS yet.  You’re not ready, my young padawan.

Enable SSL for your domains

Now we get to figure out what all those files are that certbot installed.  Remember I said there were something like 14 to 18 files generated.  Well, only four have any meaning at this point. 

1. cert.pem: server certificate only
2. chain.pem: root and intermediate certificates only
3. fullchain.pem: combination of server, root and intermediate certificates
4. privkey.pem: private key

If your next question is “what is the difference between root, server and intermediate certificates” then you’ve come to the wrong place.  I have no idea. At this point in my twisty little configurations I don’t even care.  I just want it to work.  So moving on….

These can all be found under /etc/letsencrypt/live/your.domain.com/.  Note that if you added multiple domains to a single cert then your.domain.com is probably the first one you added to the certification generation process with certbot.  Also, these files are just symbolic links to the real files.  This is so you can update the certificates using certbot, which may change the names, without having to modify your Apache configurations.  The latter just point to the files in the live directory.

Now if you’re a normal human you would probably spend some time Googling how to point your Apache configurations at these files.  There are lots articles about how certbot will do this for you.  Automatically.  As long as everything is perfect and the sun is shining and the dog isn’t farting.  But when does that happen?

There was no information I could find that worked completely correctly.  Everything says to do this.

SSLEngine On
SSLCertificateFile /etc/letsencrypt/live/www.piboxproject.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.piboxproject.com/privkey.pem

This works, but it’s not complete.  We’ll see why a little later.

Migrate web sites

So now you’ve migrated the URLs so HTTPS will work.  That’s just the first step.  The next, and harder, step is migrating your web site.  I run WordPress and wiki sites.  The wiki sites had to be manually updated by searching for references to http://<domain> and editing the files manually to use HTTPS.  Fortunately this didn’t take that long.

WordPress is a bit more involved.  You need to update your theme, your database and any custom files.  Fortunately there is a plugin that can help with this: Better Search Replace.  This plugin allows you to specify a string to replace.  So you just search for http://<domain> and replace it with https://<domain>.  You can select the tables to process.  I selected all of them.  I ran this on 5 different WordPress databases and had no problems.

But updating the themes and non-database content is all manual.  If you’ve got a modestly sized site you’re likely to find additional updates for a long time to come. 

Redirect HTTP

So Apache is configured and the web sites updated.  That’s a good start but we need to push a little harder on this and to do that we need to go back to the Apache configuration files.  Now we need to add, just above the SSLConfigurationFile entry, the following.

RewriteCond %{HTTPS} off
RewriteCond %{HTTP_HOST} ^example\.com [NC]
RewriteRule ^(.*)$ https://www.example.com/$1 [R=301,L]

These are used to map requests for any http://<domain> to your server to be remapped to the HTTPS URL instead.  This is done to deal with anything you missed in the database or other web site content. But as much as you might hope, this isn’t the last edit to those files.

Validate your work

All the updates are in place (so we think).  Time to test that you’re a good web citizen.  Go to SSLLabs site and type in your domain.  Let it run for a bit – it’s doing lots of checking.  When it’s done you get a grade.  Anything less than an A and you have more work to do. 

All the updates work but it fails validation from SSLLabs.  You get a message about the chain being incomplete and some errors related to some protocols.  The following, added just above the SSLProtocol configuration line, should take care of the protocol errors. I got these from the folks who manage my colo server so can’t say what they’re trying to do, or even if they are correct.  But they’re pretty smart dudes.  And dudettes. 

SSLProtocol all -SSLv3
    SSLHonorCipherOrder on
    SSLCipherSuite “EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSSa !RC4”

A harder problem is the incomplete chain problem.  The solution came from a non-english site, which you can decipher fairly easily.  You should also check out the Apache configuration documentation.  The gist is that we need to use a different set of Apache configurations that includes the SSLCACertificateFile entry.

    SSLCertificateFile /etc/letsencrypt/live/www.piboxproject.com/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/www.piboxproject.com/privkey.pem
    SSLCACertificateFile /etc/letsencrypt/live/www.piboxproject.com/fullchain.pem

Notice that SSLCertificateFile now points to the cert.pem, which is the server-only certificate.  And the new SSLCACertificateFile points to the fullchain.pem, which is a bunch of certificates I can’t explain.  Easy peasy, right? 

Now run your domains through SSLLabs again.  If you get an A then congratulations and yes, you’re the teacher’s pet.  If you get anything else just remember: no refunds on your tuition turkey.

What I skipped

SEO.  I didn’t care.  I don’t have a godzillion (which is a google + 1) followers.  And I’m not selling anything.  So dealing with SEO isn’t an issue for me.  If it is for you, continue rubbing your magic lantern. Something will come up.  On the flip side, since switching to HTTPS the number of visitors on my sites has gone up.  Okay, not a lot.  But at my level any bump is a mountain.

Some of the SSL configuration options I put in the domain configuration files may be more appropriately placed in the ssl.conf file instead.  I didn’t try this.  I didn’t even look into this.  By the time I figured out this might be an option I had run out of sweat and enchiladas were burning a hole in my stomach.  I opted for Rolaids instead.

Certificate renewal.  Look it up.  Basically, those certificates don’t last forever.  You need to renew them.  Fortunately, by the time you get to this is it should be fairly simple.  If you know how to set up a cron job.  If you don’t, go find that lantern again.

That’s it.  This wasn’t supposed to solve your problems.  And it probably didn’t.  But hopefully it gave you some pointers to get started at solving them. Good luck!!

The post Migrating from HTTP to HTTPS with certbot first appeared on Michael J. Hammel.

Unpacking

The printer comes ready to use, for the most part.  There is no assembly required.  Opening the box you find a styrofoam hat with some cables and tools.  The styrofoam was a very tight fit in the box.  To get the printer out I had to flip the box over and let the whole thing slide out. There is also an SD card taped into a slot in the styrofoam.

Upside down after sliding out of the box, note the square styrofoam inserts that must be removed after righting the machine.

Inside the packaging you’ll find a manual, a quick setup guide and some other paperwork.  The first thing I did was register the printer via a text message, as per the instructions in the paperwork.  I would have preferred to do it via my computer since my phone sucks.  Don’t all phones suck?  Anyway, all went well with the registration.  So I moved on to unpacking the printer. 

The printer has a big styrofoam hat and footer so those come off first.  Then there are two large square styrofoam inserts inside the print area, plus two small mostly flat styrofoam inserts under the print bed.  All of this comes out easily.  There are two additional orange plastic pieces that must also be removed.  These are labeled and easy to identify.  These are not as easy to remove and require a little careful prying.  Be careful not to yank on any wiring while doing this.

The tools provided include: a glue stick, some plastic pads for the corners of the printer, a pair of tweezers, a small screwdriver and some lubricant in a plastic tube. There is also a large flat scraping blade.

There is also a set of tools in a plastic container.  I didn’t use any of the tools initially.  I tried some of the glue stick but found it didn’t matter much in my early testing.  The scraping tools was very necessary as many of my initial trials required scraping off bad prints from the printer bed.

The packaging includes a power cable and a USB cable.  It also includes a small spool of blue PLA.  Note that the temperature setting for this PLA is printed on a label on the side of the spool.

The SD Card

Before attaching cables from the printer to a computer or power I thought it best to read the manual.  The instructions point you to the SD card which contains a video explaining the setup.  This video plays on Linux with VLC although it seemed to drop or corrupt a few frames.  That was weird, but didn’t really bother me.

The video suggests you received a thumb drive and micro SD card.  I got an SD card w/ micro SD insert.  Not a big deal, but it was the first of a number of discrepancies.  The biggest one relates to setting up the voltage for the printer.  On the back there is a switch to allow choosing a voltage level.  The video says the settings are either 230 volts or 115 volts.  The physical switch is labeled 220 and 110.  The default printed on paper covering the plug on the printer says the machine has been set to 230.  The actual default was 110.  Try not to let any of this confuse you.  If you’re in the US, set it to 110 or 115, which ever is labeled on the actual switch. 

Most of the video talks about using the provided MatterControl software.  However, that isn’t available (without recompiling a bunch on non-C code which I didn’t feel like diving into unless required) for my Fedora platform so I skipped the rest of the video.  It may have helped to watch it just to note that tasks they performed with the software.  I probably needed to do the same tasks some other way, as you’ll see later.

Don’t get confused with the voltage. In the US, it’s 110 or 115, depending on what your Robo3D actually says on the switch.

Physical Setup

I set the printer on a work table I built from a modified Ana White design.  The table isn’t quit flat and there are no feet adjustments on the printer to level it.  The printer would wobble on the table if I manually rocked it. I thought about leveling with folded paper under one corner, the Maker way.  But I actually forgot this step and when I did my first full print I noticed that the printer didn’t rock as the extruder and bed were moved.  So leveling the printer itself might not be that important.  However, see my discussion on the level the printer bed a little later.

The printer bed moves front to back (the Y axis, I believe) and the extruder moves left/right and up/down.  The bed needs about 25″ back to front, including space for the bed to move fully back and fully front.

The printer connects to a computer using a USB cable.  There is an SD card option but I didn’t try it because the goal was to share the printer with my wife.  The USB connection on the printer is on the right side near the front.  Note that the USB cable, which is about 4′ 10″ long, is probably too short if your computer is placed to the left of the printer.  You may need a USB extension in that case.

A power switch is located in the back.  I found it easier to use, and to see the small opening where the filament is threaded, if the printer was turned with the back facing me.  That may affect how you connect the printer to the computer. 

Of course the whole point of a 3D printer is to stack plastic until it looks like something you wanted.  To do that you need to feed a filament of plastic into an extruder which melts the filament and stacks it on the printer bed.  The filament is threaded through a hole at the top of the extruder.  In the Robo3D the feed hole is small and encased by a small door with springs on screws.  While it’s easy to feed into this hole the filament has to also be fed through a hole hidden inside that is apparently smaller.  It’s very difficult to feed through both holes.  After some failed prints I discovered the best way to feed the filament is to remove the two screws and springs (careful – the pop off from the springs) and remove the door.  Then manually push the filament through the smaller, now visible hole till melted filament comes out the extruder (be sure the extruder has been heated to 180C before doing this – see the section on software).  Then replace the door, screws and spring.  In my opinion this is only real drawback to this printer’s design.  It’s just not easy to see or access the feed hole for the filament.

Note that the physical setup I used included connecting the printer to a Raspberry Pi.  More on that in the Software section.

Software

This is where things grind to a halt initially.  At least if you’re a Linux user.  Follow along, though.  In the end it’s not that hard once you have the information you really need.  Much of the work I’m going to describe might be easier if you connect the printer directly to your Windows or Mac boxes and use the provided MatterControl software.  But I’m a Linux guy and the world doesn’t try to make things easy for us.  That’s why we’re so smart.

The Robo3D wants to run MatterControl for Mac.  Mattercontrol is a slicer, which is what converts a 3D model (stl) into a format (g-code) that can be fed to the printer.  The 3D printing stack is a little confusing – no wonder some people have problems getting started.  A good summary of the software stack can be found on the Robo 3D forums.  MatterControl can run on Linux.  However you need to compile from source for Fedora.  This requires Mono 4.4.  Fedora 22, which is what I’m using, is at 2.10.  The overhead to get all that working seemed hardly worth the effort if I could find something a little more off-the-shelf.

There are other slicer software solutions out there, such as Cura.  Support for using Cura directly with Robo3D is coming, apparently.  This allows Cura to not just slice but feed the g-code to the printer and track its progress.  But that setup would isolate the printer to either my wife’s Mac or my Linux box and make it less friendly to share.  Fortunately you can use Cura on your desktop, Linux or Mac, separate from the printer.  This allows you to model and slice on your desktop and just feed the g-code output to the printer through some other method.  Since my solution will be web-based to share the printer with my wife this is the optimal workflow.

The Robo3D printer has been used with Cura to print custom pieces for DragonLock, a 3D printed dungeon gaming terrain building system.  Given the photos I saw I thought using Cura would be a better option for both my wife and myself.

In the end I opted for a version of OctoPrint running on a Raspberry Pi, known as OctoPi.  OctoPrint is a web-based solution for managing 3D print jobs, including watching real-time progress.  You upload the g-code file to OctoPrint and then print to your connected and configured printer.  OctoPi is an image you copy to an SD card and then boot on a Raspberry Pi that runs an instance of OctoPrint.  Because of my PiBox project I have a few extra Pi’s lying around.

OctoPrint can be used to perform slicing of your 3D model using a variety of slicer software.  In fact the default slicer is Cura.  However, don’t try to use the OctoPi to do your slicing.  It will be slow and probably can’t handle sequential requests very well.  You’re better off doing slicing with Cura on your desktop.  There are ready-made install packages of Cura for Windows, Mac and Debian-based Linux systems.  Fedora, which is what I use, requires a little extra work.

Cura for Linux is only available in deb packaging, which is intended for use on Debian-based Linux distributions such as Ubuntu.  Fortunately the Cura .deb file can be converted using alien (these instructions work for Fedora 22 too). 

After installing the generated RPM file on Fedora 22 (at least) you need to remove the libstdc++ installed by Cura.  Cura doesn’t list Robo3D as a supported printer so you add it as a custom printer.  It will prompt for Machine Settings, of which there are several variations I found online.

Configuration

The OctoPi setup is based in some part on the Raspbian distribution built for the Raspberry Pi.  The instructions for it are brief and mostly complete.  However, I had to manually edit /etc/network/interfaces to mimic like PiBox or it wouldn’t work. They have some clever naming schemes in their boot up that didn’t work for some reason. Where they had the following

allow-hotplug wlan1
iface wlan1-raspbian inet manual
    wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

I used the standard interface naming scheme

auto wlan0
iface wlan0 inet dhcp
    wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

I also used a simplistic setup for wpa_supplicant.conf

# country=US
# ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
# update_config=1
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
ap_scan=1

network={
  ssid="Your SSID Here"
  scan_ssid=1
  proto=WPA2
  key_mgmt=WPA-PSK
  psk="Your Password here"
  pairwise=TKIP
  group=TKIP
}

Using standard formats for the interfaces file worked better, probably because I missed a step in setting up the configuration file to set the interface naming (or the OctoPi team did).

Setting the Z-offset

My first set of prints failed completely.  I kept getting a loud click.  After some time I tried to force the filament through manually and found it to be stuck.  The filament would not go through and would not pull out.   I turned up the heat on the hot end, removed the screws on the filament feed intake and manually forced the filament through to clear any clog.  Then I was able to pull it out manually.  I clipped off the end and manually fed it through again.  I adjusted the screws to be less tight than before.  This cleared the filament path but didn’t produce any prints. 

Then I got on the ShoutBox section of the Robo3D site and someone called WheresWaldo suggested adjusting the Z-offset.  There is a unique command (M565) to do this so you can set this as the default for all your prints.

I experimented with different settings, adjusting the value and starting a print run, until I got to a Z-offset of 0.75 that seemed to work.  And by work I mean print on the bed in smooth lines without globs of filament getting stuck to the extruder. But the print was only working on one side of the bed.  The other side was not always printing.  This made me think the bed was not leveled.  A little more googling and I found the real solution to the Z-offset and bed leveling problem.

The following typed in the Terminal tab of OctoPrint should level the bed.

1. Probe the bed: G28.  This tells you what the Z-Offset should be.  Just enter the command in the terminal window and watch the output for the Z offset after the extruder stops moving.  In my case it was -0.73.
2. Set and save the Z-offset:  M565 Z-0.73  This will store the Z offset in flash so you never need to set it again.
3. Level the bed: G29  This probes 9 different points on the bed based on your stored Z-offset so that the bed is leveled automatically as the print is running.

These three are manual steps required when you first power on the printer.  Although the Z-offset should be stored and reused on power up it doesn’t appear that it has any effect without doing all three steps each time.  You can tell this is the case if, as you start a print, you hear a loud click.  That click is the extruder trying to feed the filament but failing because there isn’t enough clearance between the tip and the bed.

OctoPrint

The Robo3D should be configured for use with the OctoPrint interface. The Settings->Serial Connection fields should be set to /dev/ttyACM0 and baud rate of 250000 for the OctoPi.  A printer profile should be created to set the size of the print area.  The X, Y and Z volumes should be set, respectively, to

X: 228

Y: 254

Z: 228

And set the option for Heated Bed.

This requires  : watching the progress in the g-code viewer.  Restarting a print should start by reloading the g-code viewer.  If the print starts but doesn’t set the desired temperatures then do it manually before you start the print.  I use 200C for the hot end working with the stock blue PLA provided in the packaging and 60C for the bed.

Create a printer profile to tell OctoPi the print area of the Robo3D.

Set the serial connection for the OctoPi to talk to the Robo3D.

First Print Attempts

My initial attempts at printing were all cancelled quickly because it became apparent the filament wasn’t moving.  Each attempt had to be scraped off the bed and restarted.  This was fixed with the bed leveling process.  Once I started my real first print I began to understand something that had confused me originally.

See the ghost layer? You peel it off when you’re done printing. But this should have a solid surface, not look like a sponge.

The Robo3D (or perhaps the slicing program) lays down a ghost layer, almost like a halo, on the first layer it prints.  It’s very thin and larger than the base of my print object.  Once I let the printing continue to higher layers it became clear that this was just a base of some kind.  I’m not clear yet why this is there – possibly to make removing the print easier – but don’t be fooled if you’re anxiously watching the start of the print and see this ghost layer get laid out.  Let it run for a bit and the object should start taking shape.

When the print finished the bed moved all the way to the back, which is good considering that’s the side facing me.  It made it easier to pull the object from the bed with the scraper.  I pulled the print off and examined it.  It was the right shape but a fairly unimpressive print with many holes and no solid surface.  It looked more like a sponge than the bookmark it was supposed to be.  Note that the model for this was provided as one of four models available download after registration from the Robo3D manufacturer .

Having never done this previously myself (although I’ve had a model printed for me at the local library) I concluded that the problem was with the way I sliced the model.  I checked my Cura setup and found it to be on the Light infill setting.  There are for infill settings: hollow, light, dense and solid.  So I switched to solid and tried again.  The result was actually worse. 

So while I’m now able to print I have yet to figure out how to print well.  I imagine this is akin to learning the difference between printing plain text and printing in CMYK with GIMP.  I’ll need to check on the Robo3D forums or maybe a local Maker user group to get some help. 

One other thing:  a day or so passed between the first print and the second.  When preparing the second print I did everything I’ve specified earlier and started the print.  The extruder started to position itself and printed a small part of the ghost layer and then … POP!  The filament flew out of the feed at the top of the extruder.  It had snapped off.  I had to disassemble the feed (those two screws again) to find the part stuck in the extruder.  I pulled on it by hand and it broke again leaving a smaller piece to grab.  I had the extruder heated to 200C and then used a pair of needle nose pliers to carefully pull the remainder out.  It took awhile, but I think once it got hot enough and could stretch a little it finally came out.  This is not a good sign.  Either the extruder/feeder design is flawed or the filament is bad.  I got both from Robo3D.  I hope they can fix at least one of those problems.

The snapped filament with the feeder screws removed.

The filament pulled from the feeder after it melted enough to get it out.

The second print, which should have been better than the first but was actually worse.

Tweaks

The USB camera support on OctoPi is not working at the moment.  Neither is the automatic update.  I may need to reload that software to the SD card, setup and try again.  Perhaps this is related to the bad prints, but somehow I don’t think so.

Some other suggestions I found online include an additional fan on the opposite side of the extruder and a LCD interface.  Others have gone so far as to integrate the OctoPi directly into the printer with a single power switch.  I have too many other projects on my plate to dive into those now, not the least of which is getting this thing to make a decent print.  But let me know if you try them.

Update 2020

Three years later and I’ve finally returned to 3D printing.  I’m looking to see if I can print those face shields (eg Prusa design) needed by doctors and nurses to treat Covid-19.  Before I can do that I need to make the printer actually work.  And I haven’t touched this printer since I wrote this article the first time.

A proper print from Robo3D.

After refreshing my memory on how to get started (thanks to this article, no less) I upgraded to Cura 3.6 (latest available for Fedora 29) and OctoPrint 1.4.0, which I’m running directly on my desktop now (which has 32 cores and 32G memory, so why not?).  Everything came up quickly and I was able to try printing again rather quickly but every print was worse than the first time.  I was still getting that clicking sound and the extruder would clog up.  I tried printing at higher temperatures, seasoning the nozzle, and clearing the extruder path.  Nothing seemed to help.

Then I took a look at the nozzle head.  It was sitting so close to the bed that even a piece of paper wouldn’t fit between them.  I had set the Z-offset as described in this document but that didn’t seem to change anything.  Turns out the problem was not just setting the firmware but also setting Cura’s Z-offset by adding the Z-offset plugin.  This is required because Cura was setting the Z-offset itself, irrespective of what I set in the firmware, and I needed to override it.  The plugin can be installed directly from Cura.  In Cura 3.6 and later (maybe in earlier versions) there is a Marketplace menu option.  Look under there for the Featured Plugins and go to the end of the alphabetically sorted list.  After installation you need to click on the Build Adhesion Plate setting where you’ll find an options box to enable the Z-offset.  Once enabled you will find a new field under Build Adhesion Plate for setting the Z-offset.

More importantly, the Z-offset set in Cura needs to be different than my original experiments.  After many experiments of cancelled prints, I found the setting needed to be set to Z-1.0  which causes Cura to add the following line to the gcode.

G0 F7200 X93.014 Y71.925 Z1.18 ;adjusted by z offset

So it wants to set it to 0.18, but then adds 1.0 to it from my Z-offset setting.  This is the setting I put in the firmware to, though it’s not clear if -1.0 or -1.8 should be used.  Since MatterControl still doesn’t work with my version of Linux (and nothing on earth will get me to use Ubuntu) I’ll just use Cura and always set the Z-offset there so it embeds in the gcode file. 

The result is I now have a working configuration to print the bookmarker model.  Notice the raft printed around the bookmark.  I’m not sure where that’s coming from – I think Cura may add it.  For this print, which has some flimsy pieces where the bookmarker attaches to the book pages, this is helpful for pulling the print from the bed.  You just have to cut all that off later.   Seems like a lot of waste, so I’ll need to see if I can reduce the size of the raft somehow.

But at least the prints are working now.

Update 2020.2

The excess printing is called a raft and is a configuration item in Cura, under the Build Plate Adhesion setting as Build Plate Adhesion Type.  The raft setting for this type prints extra support area for models that need it.  But using the skirt setting, instead of raft, just prints a couple of circles around the outside edge of the model.  This is done to prime the extruder before starting to print. Those extra circles are on the build plate but are not attached to the model.

The result is much better, at least for this model, as you can see in this picture.

Bookmark printed using the skirt build adhesion type setting in Cura.

Update 2020.3

I’m finally getting the hang of the printer.  Every print needs to be setup properly.  The steps I have to take are as follows.  These are all done through Octoprint.

1. Raise the print head.  I click on the Raise button in the Control tab twice.
2. Heat the head to 240 C.
3. Extrude until I see a clean stream of filament.  This flushes any dried blobs from the head.
4. Home the print head X/Y.
5. Remove the filament flush and clean the print plate.  Scrape any previous glue from the plate and wipe with a soft cloth, like a eye-glass lens cloth.  Use Windex if necessary.  Doesn’t have to be perfectly clean, but remove any large lumps that might get in the way of the print head.
6. Rub Elmers Washable School Glue Stick over the area of the print plate where the print will go.  This is a bit of a guessing game for odd shaped prints.
7. Home the print head Z.
8. Use the Terminal tab to type: G28, M565 Z-0, G29 and wait for the print head to stop moving.  In some cases it helps to change Z-0 to Z-0.1, like when you find the print head is too close to the surface while it is extruding.
9. Home the print head X/Y again.
10. Heat the bed to 60 C.

Now I’m ready to print.  It’s important that the model being printed, in the gcode, have the Z-offset properly set as this method relies on the model setting the Z-offset and NOT Octoprint!  Note that I’m using Cura to slice my models and it has presets for the Robo3D R1+.  Also note that it should be set to 210 C to print.  That’s the optimal print temperature for PLA that came with the printer.  Other filaments will have different temperatures but you can set this is Cura for each model you slice.

Finally, I thought I’d post an image showing the selection of filaments supported by the Robo3D R1+, which was taken from Amazon via Robo3D.  This is just for reference when looking for compatible filaments.

The post Getting started with the Robo3D printer and OctoPrint first appeared on Michael J. Hammel.

The room only holds 10 people so get there early to get a seat!

The post First PiBox Meetup first appeared on Michael J. Hammel.

One of the most important tasks when writing a server library is unit testing the inbound message handling.  So how can you do this?  There are many testing infrastructures, but it’s actually pretty easy to do without one.  For PiBox’s piboxd daemon I wanted to automate testing using ordinary BASH scripts.  Before getting to the shell scripts let’s define the protocol that will be tested.

Binary protocol format

Before explaining the test method I should explain what I’m testing.  There are three components to the server that need testing:

1. The inbound connection handler that validates the message
2. The queue handler that is used to asynchronously handle the message
3. The message specific functions

The server doesn’t get a lot of connections so it’s not implemented to be robust (a bug on its own).  It handles each connection and attempts to read in the complete message.  If it fails, it drops the connection and waits for the next message.  If the message is complete it queues it for a separate thread to handle.

The incoming message is a self defined message.  This means it has a small header with a payload size and the payload.  The header is defined by the message protocol and contains a type and action.  If the payload (or any other part of the message) does not arrive in time then the connection times out.

The header has to be validated for the protocol.  If it fails the connection drops without reading the payload.  The payload depends on the message type and action.  The queue processor passes the message to its handler.  If necessary, the inbound connection is kept open and passed along to the message handler who is responsible for cleaning up the connection.

The payload is shown with an alternate format that is used for specific commands where a tag and a filename are used.  The use of the tag depends on the header where specific types require the tag and filename.

As you can see there isn’t much to this protocol.  It’s pretty simple though there are a number of points of potential failure.  Much of the protocol is handled in the header to perform various network setup.   Configuration data is in the payload.  Unknown types and actions are ignored.

Now that the design is defined it’s time to look at unit testing the inbound, queue and message handling.

Send a packet with netcat

There are probably other ways to do this (it’s software, there are 1000 solutions with at least half being sufficient) but I use netcat.  The command line for all tests is simple:

nc -4 localhost $PIBOX_PORT < servertest.dat

The -4 option means use IPv4.  Doing the tests on localhost is easier if I limit the protocol to IPv4.  I’m not testing network connectivity, just the application layer protocol.  The port depends on how I run the piboxd daemon.  The redirection from stdin is used to pass the data to transfer to piboxd.  It’s the protocol packet we’re testing.

Build a packet with xxd

The secret to protocol testing with netcat is building the packet to feed to it.  The packet defined above uses single byte binary fields which are not easy to generate with ordinary shell scripts.   Fortunately they are very easy to generate with xxd using a three part command.

printf "0: %s" $header | sed -E 's/0: (..)(..)(..)(..)/0: \4\3\2\1/' | xxd -r > servertest.dat

The header value is a string representing the four bytes of the header in the protocol packet.  The string is represented in LSB first, as in

00000201

This makes it easier to map to the protocol diagram.  But on an Intel box the ordering needs to be reversed.  This is accomplished with the sed command, which translates the string to this.

01020000

But this is a still a string.  In order to pass it with netcat we need to make this a binary value.  That’s where xxd comes in.  Normally xxd is used to convert binary data into a human readable string.  Using the -r option we can convert the string back into binary and send it to a temporary file, the one we’ll pass to netcat to run our test.

This is just an example of generating the header but the same process is used to generate the payload size and payload fields, appending them to the temporary file.

Scripting the whole thing

To script the tests I used two functions:  genMsg and sendIt.  The former is used to generate the temporary file used to run a test.  The latter is used to send the message from the temporary file.  Simple enough.  In practice these functions used state variables to modify the message content in order to apply invalid formats.  But a simple test might look like the following.

genMsg 00000201 webcam
NCOUT=1
sendIt
unset NCOUT

The genMsg function is handed the header and a payload.  The NCOUT variable is used to save output from netcat to check for error messages.  As the script expands many such variables are created to vary the tests.

Extensions

As more features were added to piboxd more unit tests were added to the script.  This includes the ability to launch an netcat-based web server to receive replies from piboxd as well as sending and receiving multicast messages using socat (an alternative to netcat).  I was even able to run piboxd with valgrind to check for leaks.

The scripts for testing piboxd are in the tests directory but need to be run from the top of the source tree in order to find required data files.  Feel free to check them out and let me know if you have additional ideas!

The post Unit testing server protocols in C first appeared on Michael J. Hammel.

I keep a page in Project Cody with ideas for future projects.  This past year I collected a bunch of ideas I want to work on, most of which are related to extending PiBox in one manner or another.  Some are related to protecting my web sites and servers. This doesn’t include continued work on PiBox issues nor does it include my plans for adding IoT support for managing devices via wifi.  The IoT projects are part of my IronMan project – the idea that I can build Iron Man’s house using existing technologies.  It’s not really much of a stretch.  The real issue is getting glass that will act as a display, either directly or by electrically opaquing the substrate. 

I’ve been working on ideas for Iron Man for some time.  I already have the Jarvis project for accepting voice commands and evaluating them for basic control of the house by simply passing audio files to Google’s voice to text translator (though that may not be the one I have implemented in Jarvis now) and then passing the text through a natural language processor to interpret the command.  There really isn’t much to that project now except a proof of concept.  And it doesn’t have Jarvis’ voice.  Generating a realistic voice is a fairly time-consuming activity and I’ve so many other things that take precedent.  But the basic structure is there.  Extending it to do something useful won’t happen till I have some devices to control.

Which leads me to some of the technology plans I have for PiBox in 2017.  Take these with a huge grain of salt.  I fully expect my aging pups to occupy much of my time in 2017.

Custom smartphone based on Raspberry Pi

I’m so frustrated with the bloat on my Android Phone.  It worked fine when I bought it but automatic updates have pushed it to its limits, making most apps fail to start or crash after they do.  It’s not like I have many apps on the thing.  I don’t use the phone for anything other than the occasional text with my wife.  You can blame the bloat on Android or the provider (Virgin Mobile) if you want.  But the truth is that it makes sense for phone vendors to bloat the OS to force you into the next model.  It’s a trick they learned from the PC world.  And it’s crap.  We should be able to build our own phones.

Fortunately, it’s possible to build your own phone.  What’s important about this project is that

• it uses a Raspberry Pi Model B or B+, which is compatible with the Pi Zero, meaning I can improve the design.
• it uses a Raspberry Pi, which means I can port PiBox to it.

I’ll need to add touch screen support but I’d already experimented with that previously so it shouldn’t be a major effort.  My launcher app will have to understand the size of the display in order to scale icons appropriately.  It doesn’t do that now, but I think I know how to do it.  And I’ll need to write all the basic phone apps:  dialing, contact list, text messaging, etc.  None of these are particularly hard.  And no, I won’t be using scripting languages.  It’ll all be done in C.

This isn’t a high priority, but it would be an interesting update to the launcher to see the same apps on a phone as on a TV or desktop monitor.

Update: 2017-03-09

I just finished adding display resolution identification and touchscreen support to libpibox, the launcher and a couple of apps.  The library supports 9 touch regions or absolute coordinates.  So the launcher works by touching an app icon while the apps use navigations (next, prev, pause, stop, etc) through the touch regions.  I’ve also added the much needed scaling capabilities to the launcher to allow it to fit on just about any screen size.  This support may not be fully sufficient for the small screen of a phone, but it works great on the 7″ official Raspberry Pi touchscreen!  And it may be enough for any small screen.  Caveat: the launcher still requires X.org, so really tiny screens might not work well.

Update: 2017-10-03

Still not working on this (because of the job change/move to Boulder-area issue) but I found this today:  ZeroPhone.  Someone beat me to the project, but that’s okay.  I’m still interested in building my own.  And putting the PiBox UI on it for ease of use, probably with a larger screen (like the original model).

Echo Pi

Echo Pi is a project I want to work with Jarvis (my experiment with “AI” software). It’s really just an omnidirectional microphone that can be used to acquire audio to feed to Jarvis.  Put one of these in each room and I can talk to the butler without screaming.

Bluetooth audio transmitter and receiver

I want this so I can play audio in my trailer from PiBox without wires.  I want to route audio to either the audio port or to a Bluetooth transmitter.  This doesn’t solve my problem of routing the audio output from my XMPCR since it has to first be routed into the Pi.  But it’s the end goal and it’s something I’ve wanted for some time.  We can use wireless headphones while we watch outdoor movies when camping.

Smartwatch

It was only a matter of time before smart phones went the Dick Tracy route, or as old dudes say, like Space Ghost.  We don’t quite have Mr. Ghost’s three button power bands yet.  So a smartwatch (who makes up these names?) is as close as we get.  This actually looks pretty simple.  It’s unclear what software I’ll put it on it, however.  A Pi Zero might fit but may have too high power requirements.  I don’t know if I can port PiBox to an AVR (though I did something similar with the AVR toolchain once before). 

IoT controllers

This is a high priority project for me.  The goal is to extend PiBox to home automation, becoming a central controller (or distributed set of controllers) for managing my home.  It starts with simple motor controls for blinds and vents.  But there is no reason it can’t also manage the thermostat.   Eventually I want to be able to schedule light switches and outlets to power on and off based on my schedules.  This will eventually replace the home automation system I have now and finally get rid of the horrible management interface they provided.

DIY Segway

So my employer, WD, is moving to new offices this spring.  The current office is about a 6 minute commute from my house down fairly busy streets.  The new office is about a 3 minute commute and I only have to cross one busy intersection.  Seems a waste to drive that far, but walking isn’t an option given Colorado winters.  So what’s a techie to do?  Build a Segway, of course.  It just needs to be a bit all-wheel drive given the route I have to take, but I think this will just meet those requirements.  I just worry about the stability of the software.  In this project, I’m less likely to be working on that than the others.

Onion Pi

This one isn’t really that important since I can do the same thing right on my laptop, but it might be interesting to see if I could put it in PiBox.  I’m not sure why I would since browsing the Internet is not supported in PiBox (WebKit takes forever to build), but you never know. 

The Onion Pi is a Raspberry Pi with Debian acting as a Tor proxy.  The Tor component anonymizes (mostly) your online browsing.  It doesn’t do much to prevent access to your computer (try running a firewall and use ipset to block ranges of IP addresses for that) but it’s not a bad idea to keep people from snooping on your obsessive and highly unhealthy need to know what the Kardashian’s are doing.

Two Factor Authentication everywhere

This isn’t a PiBox thing.  It’s a protect-my-systems-from-bad-dudes thing.  I just want it because, well, everyone is pushing it and it kinda seems like a good idea.  Except for having to use my smartphone to login to my Linux boxes.  That seems sooooo counter-productive.  I really hate smartphones.  Except for the one I’m going to build myself, of course.

HTTPS for all web services

This is something I haven’t done yet because I don’t do financial transactions on any servers.  But I hear the entire net is migrating into HTTPS so I guess I better do something for my array of domains.  It doesn’t seem particularly difficult given the use of certbot.  But working with keys and certificates has always been a bit annoying to me.  Perhaps because I just hate the idea that someone doing mischievous things forces me to do something I didn’t feel like doing.  Bad guys are annoying.  Can’t we all just get along?

IPSet to block the annoyers

I also plan on improving my internal security net for my home and public servers.  I’ve already added ipset to block large IP ranges which has had a noticeable improvement in my public web servers.  It also makes it pretty easy to block addresses trying to crack our email server.  I’m sure I can’t keep them out.  But I can make it particularly annoying to try.

And then there is…

I’ve also got notes on working on water harvesting for my garden and power harvesting for off-grid PiBox.  But those still need better plans.  I’d also like to do more work with solar power to charge batteries and hydro– and aquaponics.  I’ve experimented with hydroponics in over-winter gardens but these were mixed with plants from outside that were brought in, which led to bug infestations and a failed project.

And my back yard DIY vegetable planters needs to be migrated across the back yard so they’re out from under the now-too-tall trees and into the real sunlight – at least for the three-month growing season here in Colorado.  And I need a new potting bench.  And my wife wants new planters in the front yard.  And the fence needs mending.  And the porch needs painting.  And the side yard needs to be re-graded.  And the railroad ties in window wells are rotting and need replacing.  And ….

The post What to do in 2017 first appeared on Michael J. Hammel.

The post PiBox @ Colorado Springs Open Source Meetup first appeared on Michael J. Hammel.

The original clock was primitive, mostly because I didn’t understand how to use images and transformations correctly, even though I’d used them in the PiBox launcher!

Having just completed work on a wifi scanner for PiBox Network Config, I decided to expand on the lessons learned with Cairo to add themes to PiClock.  The original clock was a simple drawing of the clock face boundary with three hands.  The biggest task there was to compute the end points of the lines drawn for each hand.  Nothing complex there.

But there is no reason custom clock face images and hands can’t be used.  The trick is learning to work with Cairo layers (which happens without extra coding) and discovering how to properly rotate and position the hands.  Turns out none of this is hard, but it was hard finding out how to use the Cairo API.

So the first thing to do for a theme is define what I wanted to display.  There are five images you define for PiClock themes:  the clock face, the hour, minute and second hands and an overlay image.  The overlay allows you to add special effects like a glass cover.  The theme needs a configuration file to specify these images.  I chose to use a JSON format.  Previously, PiBox apps used XML for configuration files but in the last year I’ve become a big fan of JSON due to its easy-to-read name/value pair format.  Also, I’ve found a very simple to use and lightweight JSON library called Parson.  I’ve integrated this into libpibox so all apps can use it. 

The JSON format for PiClock themes is extremely simple.

{
     "theme": {
         "face":"<Name of clock face file>",
         "overlay":"<Name of clock overlay file>",
         "hour":"<Name of hour hand file>",
         "min":"<Name of minute hand file>",
         "sec":"<Name of second hand file>",
         "center":{
             "hour": { "x":n, "y":m },
             "min": { "x":n, "y":m },
             "sec": { "x":n, "y":m }
         },
     }
}

The clock face has a dimension limit of 512×512 which the code enforces.  After experimentation on the PiBox hardware I found it was best to make faces at that size and not smaller, but I don’t enforce that.  The center object defines the rotation point within each of the hands images as it cannot be assumed the hand rotates around its real center.  Other than that, the theme is self explanatory.

Before I could parse the file I needed it to have a home.  I use the -T option when testing PiBox apps which points the code at files within the source tree.  In production the data files are stored under /etc/piclock/themes.  Since there can be more than one theme there needs to be a top level file stating the currently selected theme. 

Parsing the file requires a single line of Parson code, plus extracting a reference to access fields.

JSON_Value data = json_parse_file_with_comments(buf);
obj = json_value_get_object(data);

The data object is abstract.  Fields can be extracted rather easily using the dot functions:

face = json_object_dotget_string(obj, "theme.face");
hour = json_object_dotget_string(obj, "theme.hour");
min = json_object_dotget_string(obj, "theme.min");
sec = json_object_dotget_string(obj, "theme.sec");
overlay = json_object_dotget_string(obj, "theme.overlay");

This extracts the filenames, which must be prefixed with the installation directory.  Following this is an in depth evaluation of the configuration to make sure files exist, can be read into Cairo surfaces and have reasonable sizes with centers properly defined.

A clock theme is drawn by first drawing a black background (for any transparent regions in the images).  The clock face is drawn over this.  The surfaces for the face and hands are generated when the theme is read so that they can be reused every time the clock is updated.  The hands are drawn over the face and the overlay over the hands.  Those are the layers.  As long as they are drawn in this order the layers are composited correctly.

Now for the important trick:  translation and rotation.  What was difficult to discover the use of patterns and matrices for the drawing operation.  The pattern is effectively a copy of the original image surface.  The pattern is first translated so its defined center is at the origin (0,0) at which point the rotation is applied.  Then the pattern is moved to the center of the clock window.  Obviously the face and overlay don’t need rotations, just the hands.  Here is the generic handler for hands.

void draw_hand(GtkWidget *widget, cairo_t *cr, gint hand)
{
    cairo_status_t  rc;
    cairo_surface_t *image = NULL;
    cairo_pattern_t *pattern;
    gint            width, height;
    gdouble         offset_x, offset_y;
    GtkPiclock      *piclock;
    GtkRequisition  req;
    cairo_matrix_t  matrix;

    piclock = GTK_PICLOCK(widget);
    req.width = gdk_window_get_width(widget->window);
    req.height = gdk_window_get_height(widget->window);

    /*
     * Setup based on the hand requested.
     */
    switch( hand )
    {
        case HAND_HOUR:
            image = piclock->theme->hour_sf;
            degrees = -1*(double)piclock->hoursHand/60.0*360.0;
            offset_x = (double)piclock->theme->hour_pt.x;
            offset_y = (double)piclock->theme->hour_pt.y;
            break;
        case HAND_MIN:
            image = piclock->theme->min_sf;
            degrees = -1*(double)piclock->minutesHand/60.0*360.0;
            offset_x = (double)piclock->theme->min_pt.x;
            offset_y = (double)piclock->theme->min_pt.y;
            break;
        case HAND_SEC:
            image = piclock->theme->sec_sf;
            degrees = -1*(double)piclock->secondsHand/60.0*360.0;
            offset_x = (double)piclock->theme->sec_pt.x;
            offset_y = (double)piclock->theme->sec_pt.y;
            break;
    }
    if ( image == NULL )
        return;

    width = cairo_image_surface_get_width(image);
    height = cairo_image_surface_get_height(image);
    pattern = cairo_pattern_create_for_surface (image);

    /* translate the rotation point of the hand to the origin. */
    cairo_matrix_init_identity (&matrix);
    cairo_matrix_translate (&matrix, (double)offset_x, (double)offset_y);

    /* Rotate to the correct clock position. */
    cairo_matrix_rotate (&matrix, deg2rad(degrees));

    /* Move to the center of the clock window */
    cairo_matrix_translate (&matrix,
            -req.width/2,
            -req.height/2);
    cairo_pattern_set_matrix (pattern, &matrix);

    cairo_set_source (cr, pattern);
    cairo_pattern_destroy (pattern);
    cairo_paint(cr);
}

The test theme has a reflective overlay appliedover the face and hands.

After getting the window dimensions and accessing the previously created image surface, a pattern is created and a matrix initialized.  Then the translation is done to move the configured center point to the origin.  A rotation is applied and the pattern is moved to the center of the window.  Once the pattern is set as a source for the main Cairo surface it can be destroyed and the main surface updated.

There are now two themes for PiClock.  The first is a test theme that includes the overlay.  This isn’t as clean as I’d like – the overlay doesn’t quite look like a reflective surface to me.  The overlay is made from a couple of white to transparent radial gradients made with GIMP.  Cairo composites transparency quite nicely, as can be seen with the clock hands. 

Once the test theme was working I had to make one more theme, to prove that the design worked in general.  So I created the roman theme, which is simply an old clock face with roman numerals.  The hands were run through GIMP a few times to clean them up and remove colored areas around the hands that were an artifact of the poor original image. 

The roman theme doesn’t use an overlay, but the overlay image is required because I didn’t go to the length of allowing it to be optional. The the image is simply the size of the clock face image but is completely transparent.

The theme design worked perfectly on the second theme.  I created it to spec and updated the default theme name, then ran PiClock.  It displayed as shown.  No code changes were required.   I’ve got some videos of the clocks running, if anyone is interested in watching a clock ticking.

It’s not perfected quite yet, however.  I need to add an option to cycle through themes.  But its the holidays and I’ve already spent far too much time on PiBox so it’s time to let it go a bit and get back to the things that matter.  At least for awhile.

The post PiBox: PiClock themes using Cairo first appeared on Michael J. Hammel.

Flash forward to Halloween 2016 and the Maker Faires in Colorado Springs.  I got an itch to add some kind of audio playback device to go with the video player for PiBox Media Center.  I have lots of audio ripped at home but it turns out writing a database handler for the ID3 tags was harder than doing the video lookups on TheMovieDB or the TheTVDB.  So I’ve punted on playing my music for now.  But the XMPCR’s were still around in a box in my office.  So I dug them out and started digging in.

PIXM

PiXM is an XM Radio app for PiBox. It manages an XMPCR connected over USB to the Raspberry Pi.

The first thing I did was pull out the old Perl scripts.  There is little support for scripting in PiBox because scripting languages are not my thing.  Oh, I use them plenty.  But I want PiBox Media Center to show what you can do with plain ol’ C.  Scripting is for kiddies.  PiBox is for grown ups.  Anyway, the perl scripts still work after all these years and I was able, after re-signing up the devices with SiriusXM, to use them to listen to pretty much any channel with the device connected to my laptop (not PiBox).  The devices have an ordinary audio plug for output – there is no playback through the PC.  But that’s not a major issue to me.  I can always route the output back into the Pi and manage it that way if I really need to.

So the devices still worked.  Now I needed to look at my old GTK+ code.  It also worked.  Badly.  Not because the libraries had changed but because it wasn’t a well written app and wouldn’t fit into the minimalist design of the PiBox Media Center UI.  So I decided to start over.  And the first thing I wanted to do was write my own serial protocol management to avoid calling Perl scripts.  After all, the device’s serial protocol was very simplistic and there really wasn’t much code to port.  There were even some example implementations.

Work on the protocol went quickly and I was able to get data into and out of the device without much trouble.  The next step was to implement a UI.  The UI needed to adhere to the same basic usage principles of the video player which uses just the arrow, tab, ESC and ENTER keys for navigation and selection.  This wasn’t as hard as it seemed.  I created two lists, one for categories and one for channels.  Choosing a category showed the list of channels in that category.  Tabbing between category and channels would display all channels.  Searching used the same builtin GTK+ mechanism for trees (just type some letters).  Easy peasy.  Unlike the video player, however, there are no graphics to display on the right side of the UI.  Instead, it’s just a set of four fields for channel, category artist and title.

In testing this layout I discovered some basic problems.  The first was that the serial device, /dev/ttyUSB0, is not accessible by default.  I’m using mdev in PiBox and didn’t set it up properly to give non-root users access to the device.  But before I remembered that was the problem (the app is not privileged so runs as a non-root user so I need to fix the mdev setup scripts)  I added a device state icon to the menu bar.  This bar is drawn with Cairo and is where you see the tab options for Category and Channels.  The device state icon is green for available and red for unavailable.  Makes it easier to identify hardware problems.

PiXM Controls: The Categories and Channels are selected using the Tab key. The device status icon is green, showing the device is available. The Mute icon is not set. Pressing M will toggle the mute state of the device’s audio output.

I also noticed that the device serial protocol allows for muting the output.  Seemed like it might be nice to add that feature, so I added one additional key, M, to toggle the mute on the XMPCR.  This violates my desire to keep all PiBox Media Center UI components (other than the PiBox Network Config app) to just the arrows, Tab, ENTER and ESC key.  But since PiBox is using those nice FAVI keyboards, I figured what the heck.  Whose gonna be annoyed?

The majority of the app is plain ol’ GTK+ 2 widgets.  The control bar is a drawing area with Cairo doing custom rendering.  I use Cairo for much of the PiBox UI: PiClock, the video player (aka VideoFE) and the launcher use it extensively.  Like the video player, the PiXM app does simple on-demand drawings in the drawing area.  This isn’t exactly efficient or by design of GTK+.  After working on the wifi scanner (see below) I realized I should have written my own widget to better deal with async issues inside the main GTK+ loop.   PiClock has such as widget, but it needs additional work based on lessons learned on the wifi scanner.

Wifi Scanner for PiBox Network Config

The wifi scanner is implemented as part of the PiBox Network Config app. It should probably be set as optional since the app can be used to configure wired ethernet too.

One of the major lessons learned from the Maker Faire was a need to solve my problems with PiBox acting as an access point for a local network of PiBox players.  At home I had a setup of three PiBox Players playing movies from a PiBox Media Server.  This worked fine there.  But taking the same setup to the library (and Barnes and Noble for a separate Maker Faire event) showed that wifi performance was unacceptable. 

I’ve done lots of research on possible causes for poor wifi performance.  I first thought it was a bad Samba configuration since videos are accessed by players from a Samba share on the server.  That helped, but didn’t solve the problem.  Then I found that images used in the video player browser were huge.  So I scaled them back and that helped a bunch with browsing the list of available videos.  But it didn’t solve the issues related to video playback.

So then I dug into wifi itself.  I found that channels 1, 6 and 11 are typically the default channels in routers because they apparently don’t overlap other channels as much the other channels overlap each other.  Also, signal strength is important since choosing a channel where your signal strength is better than other networks on the same channel will help with throughput.   Choosing a channel with fewer other networks using it requires knowing which networks are using which channels.  Same it true for signal strength.

Which brings me to the discovery of Kevin Yuan’s Wifi Analyzer for Android.  What a cool little app.  And that’s coming from a guy who doesn’t use ANY apps.  It shows an animated graph of wifi networks in the local area.  So I looked into how you get the data for this kind of graph.  Turns out you can get everything you need by looking at /proc/net/wireless and using iwlist and iw.  More easy peasy.

The scanner shows the set of SSIDs along with their channels and signal strengths as text on the right. The graphs are filled with semi-transparent coloring so the text can be read even of the graphs extend into the text area.

With the data gathering solved, I needed to integrate a UI and a way to get the gathered data into the UI.  One problem to solve is how to choose separate colors for each network.  In the end I used the golden ratio.  Next is how to get the data, which can take several seconds to acquire, without interfering with the UI.  So I put the data gathering into a background thread that made calls to the pnc (PiBox Network Config) library.  The data was stored in that thread and was made available to the UI using mutexes. 

I first made the UI act like the control bar of PiXM but that didn’t work.  The periodic updates caused lockups in the GTK+ main loop.  So I switched to writing my own widget, just as I’d done with PiClock.  This widget is far more complex and in doing the port I found bugs in PiClock.  But using a widget solved the problem of GTK+ main loop lockups. 

Drawing the graphs is done with Cairo.  If you look at the graph it looks like a bunch of parabolas.  At first I looked into how to graph these.  But the parabolas needed to be closed on the wide end in order to fill them.  That’s when I realized the trick to graphing in Cairo:  using masks.  There was an example of how to create an ellipse by using transforms to fit the ellipse in a box.  The transform placed the center of the ellipse on the channel axis with the left side drawn off the left edge of the widget, making it only partially but annoyingly visible.  The fix to that was to mask off the area to the left of the axis and THEN draw the axis.  Cairo layers the drawing commands so the mask cuts out the left side of the ellipses and then the axis is drawn over the mask.  More easy peasy.  After many hours trying to find the trick.  Silly me.  It was in the documentation.  The last place one would look.

You might wonder why the graphs are drawn left to right instead of bottom to top.  Simple:  space.  I have lots of vertical space here but not much horizontal space.  So I just draw left to right.  Seems okay to me.  Comments welcome.

What’s Next

I need to return to PiClock and clean up it’s widget, adding custom layers to the Cairo rendering to allow for custom themes for the background, hands and overlay.  The same needs to be added to the launcher so I can do things similar to what you see with the Roku front page.

The post PiBox: XM radio app (PiXM) and wifi scanning in PNC first appeared on Michael J. Hammel.