Postbox: Now Worse than Microsoft

Normally I use this blog exclusively for technical articles but in this case I will be mixing a little tech with a little op-ed.

A while back I switched my E-mail client to Postbox, which is derived from Mozilla Thunderbird.  It wasn't free, but I didn't mind paying for something with a little more polish and presumably, support.  I had been happy with it for the time I used it, until recently when it automatically updated to version 4.  Cool.  It had a more mobile app aesthetic as is common for a lot of applications these days, and supposedly had a bunch of improvements and features.

What I didn't know at the time was that you have to pay for this upgrade.  No, having an existing license for the previous version does NOT unlock the new one.  I found that mildly annoying, since I use several paid-for applications that allow me to continue to use my license to receive upgrades.

So whatever... I was happy with Postbox 3 so I didn't worry too much about it, hoping that when the "trial period" ended I could just downgrade back to the version I had already paid for.  Kinda like how if, in the past, if you didn't want to pay to buy the newest Windows version you could just stick with the one you already had.  (You still can, if you don't like being spied on.  I am still sitting on Windows 7 thank you very much.)

Guess what?  That isn't an option.  As of this morning, I can no longer use Postbox to send or receive E-mails unless I pay for the upgraded license.  If I had kept my original installation .exe I could perhaps have reinstalled the old one with the license, but I generally don't keep that stuff on my system to clutter it up because USUALLY if I need to roll back I can always go online and get the older version downloaded again.

So here I sit, unable to use software I paid for.  Looks like I'll be importing my E-mails into one of the free E-mail clients, like Thunderbird.

Before I get to that though, I have some remarks for the people at Postbox.  I notice their "Contact Us" page doesn't seem to have a simple mechanism for sending them some feedback, though they do provide a snail mail address for "Fan Mail."

A bit full of ourselves, are we?

I do find it telling that they don't seem to want to hear directly from their "fans" through some kind of E-mail mechanism.  Anyway, here's what I have to say.

Dear Postbox Management and Development team,

I really have to hand it to you.  You have managed to make Microsoft, the company with one of the most evil reputations in the IT industry, look reasonable.  I can understand how a small company that doesn't have a ton of operating capital might need to charge for major updates to their product.  I get that, but what you've done here is to eliminate choice.  As a user of your product, I'm being forced to pay you to continue to use software that I already paid for once before.  As far as I can tell, I've never had the option to downgrade back from the trial version of Postbox 4, and if you included some kind of warning before I installed the version 4 update, I didn't notice it.  I've read articles in the past about the reputation your company has with its customers, and what I'm seeing here does seem to support the idea that you really don't have much regard for your customers, only their money.  On top of that, you have the gall to call incoming correspondence "fan mail."  I don't know whether to laugh out loud or just shake my head ruefully.  For you, I think I'll try to do both simultaneously.  

It isn't even like your product is great enough to justify paying for every major upgrade (or buying the "lifetime license" I've heard about.)  As someone who has used both your product and Mozilla Thunderbird, I don't see a dime's worth of difference between the two, only Mozilla gives Thunderbird away for free.  You know Thunderbird, I assume, as it's the basis for Postbox in the first place.  If I recall correctly, I purchased Postbox originally for $10, and now you want another $15 for the privilege of continuing to use the application.  Your site talks about all of the wonderful new features of Postbox 4, but I never noticed much of a difference except for a rather more awkward and Romper-Room looking interface.

In any case, I won't be paying you again for the same software, whatever your claims are about the superiority of version 4.  You got me once, it won't happen again.  Frankly I never saw much advantage to Postbox over Thunderbird in the first place, and had only been using it because I'd already paid for it.  I will close this message now, but I won't wish you well.  I think companies that do business like yours does deserve to fail.  Instead, I'll wish you wisdom, that you might realize how this alienates customers and will make good adjustments in the future.  

Ok, with that out of the way, let's talk about migrating to Thunderbird.

First, don't bother with the instructions on Postbox's support page.  As of the time I'm writing this, they're from 2009 and not only use basically the same procedure as if you were using Thunderbird already (remember, Postbox is Thunderbird with tweaks.) but they also even provide a link to a Mozilla support page for additional information.

It's really just a simple mater of using Import and Export tools from within Thunderbird to import your E-mail folders from Postbox.  You'll find them in Windows in AppData/Roaming/Postbox and AppData/Local/Postbox.  I haven't quite got the hang of managing E-mail folders so I can't offer any advice there, but at least this will enable you to save your messages so you can safely get rid of Postbox if you want to.

Querying OpenTSDB with Java. Yes, it can be done!

We needed a Java client for OpenTSDB to store our time series data and so I'll share the result here.

Below, I'll be fleshing out two ways of storing data in OpenTSDB using the HTTP API with Java.  The first is using POST, where the payload is a JSON Object containing all of our query parameters.  The second way uses GET and all of the query parameters are in the request string.

Let's jump in.

We'll be using OpenTSDB version 2.0.1 today.

So for starters, let's do the POST approach.

First, open your HTTP connection in the usual way.  Make sure it's set to POST and you're setting DoOutput and DoInput.

URL url = new URL(urlString + "/api/query/");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty("Content-type", "application/json");
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);

Now get yourself an OutputStreamWriter.

OutputStreamWriter writer = new OutputStreamWriter(httpConnection.getOutputStream());

At a minimum, OpenTSDB requires a start time, end time and metric for querying data.  The start and end times should be expressed in epoch time using a long.  It's also a good idea to specify the aggregator you want to use although that isn't required.

long startEpoch = 1420088400L;
long endEpoch = 1425501345L;
String metric = "warp.speed";
String aggregator = "sum";

I'll also add a couple of tags in order to show how to do it below.

So now, we create our main JSONObject.

JSONObject mainObject = new JSONObject();
mainObject.put("start", startEpoch);
mainObject.put("end", endEpoch);

So far so good.  Next, we need our query parameters.  This includes the aggregator and metric.  We will put these in a JSONObject which, in turn, will live inside a JSONArray.

JSONArray queryArray = new JSONArray();

JSONObject queryParams = new JSONObject();
queryParams.put("aggregator", aggregator);
queryParams.put("metric", metric);

And we put this into the array.

queryArray.put(queryParams);

Now, if you're using tags, add them next.

JSONObject queryTags = new JSONObject();
queryTags.put("starshipname", "Enterprise");
queryTags.put("captain", "JamesKirk");
queryTags.put("hullregistry", "NCC-1701");

And add that to the queryParams object.

queryParams.put("tags", queryTags);

Now add the queryArray to the main object.

mainObject.put("queries", queryArray);

You can now write this JSONObject as a String to your connection.

String queryString = mainObject.toString();

writer.write(queryString);
writer.flush();
writer.close();

Now, the server will respond with a response code.  If it's good (HTTP_OK) then you should be able to get the response data.

int HttpResult = httpConnection.getResponseCode();

if(HttpResult == HttpURLConnection.HTTP_OK){
     result = readHTTPConnection(httpConnection);
}

And here's the readHTTPConnection method:

public static String readHTTPConnection(HttpURLConnection conn){
StringBuilder sb = new StringBuilder();
BufferedReader br;
try {
br = new BufferedReader(new InputStreamReader(conn.getInputStream(),"utf-8"));
String line = null;
while ((line = br.readLine()) != null) {
sb.append(line + "\n");
}
br.close();
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return sb.toString();
}

What you get back is a string that can be turned into a JSONArray.

return new JSONArray(result);

Now, that's the way I like to do it because working with a bunch of String appenders feels clunky to me, but if you're a fan of that approach, here's how you can run that same query using an HTTP GET request.

long startEpoch = 1420088400L;
long endEpoch = 1425501345L;
String metric = "warp.speed";
String aggregator = "sum";
String result = "";

So now we manually build our request string:

StringBuilder builder = new StringBuilder();

builder.append("?start=");
builder.append(startEpoch);
builder.append("&end=");
builder.append(endEpoch);
builder.append("&m=sum:");
builder.append(metric);

Now, if we have tags, they should be in pairs and the entire set is enclosed in curly braces and separated by commas.

builder.append("{starshipname=Enterprise,captain=JamesKirk,hullregistry=NCC-1701}");

URL url = new URL(urlString + "/api/query/" + builder.toString());
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty("Content-type", "application/json");
conn.setRequestMethod("GET");

And we get the result data back exactly the same way we did above.

int HttpResult = httpConnection.getResponseCode();

if(HttpResult == HttpURLConnection.HTTP_OK){
     result = readHTTPConnection(httpConnection);
}

return new JSONArray(result);

So the GET approach is shorter but I personally prefer the POST method.  Either way will get you the same results.

Storing Data in OpenTSDB Using Java

Not a lot of examples exist for showing how to use Java to store a time series in OpenTSDB.  It isn't complicated, but there are a couple of noteworthy items to keep in mind.

The OpenTSDB I'm using is running on a 3 node distributed cluster built on the following:

CentOS 6
Apache Zookeeper 3.4.6
Hadoop 2.4.0
HBase 0.98.8
OpenTSDB 2.0.1
Java 7

This post assumes you have a working OpenTSDB instance.

The first thing to understand is that the OpenTSDB HTTP API is what makes it universal, in terms of what languages you can use to build an interface with the database.  Ultimately what gets passed into the OpenTSDB URL is a JSON array containing all of the time points to be stored.

In this example, we're storing ECG time series data for an anonymous patient.  Due to HIPAA regulations, even the specific date the data was gathered is off limits, so we will arbitrarily choose 12:00 AM on 1 January 2015 as the start time for this time series.

Step 1: Create an ArrayList of dumb data objects to represent the time series.  Each instance of the DDO will represent a single time point.  There is an example of a suitable class on OpenTSDB's GitHub repository.  The class should have, at minimum, a field for the timestamp, a field for the data value, the metric to be stored and a series of tags.

The metric is the unit being stored.  In an ECG time series, for example, the value is microvolts.  Since ECGs store data in multiple simultaneous channels, I am creating a separate metric for each.

NOTE:  Before you can store values using a particular metric, you must register that metric in the database!

Use the command mkmetric to register your new metric.  For example:

./tsdb mkmetric ecg.V6.uv

where ecg.V6.uv is the new metric being created.

In my own version of the IncomingDataPoint, I changed the type for the field "value" to an int.  This resulted in the resulting JSON array looking the same as in OpenTSDB's documentation.

Timestamps should be expressed in epoch format when stored in the IncomingDataPoint.  In the case of my time series, the data comes in at variable sampling rates, usually around 500 Hz.  OpenTSDB can store time series in intervals as small as one millisecond, which is sufficient for this rate.  The epoch value for 12:00 AM 1 January 2015 is 1420088400000, with resolution to the millisecond.

Tags are an optional way to add details to individual time points that can be used in searches for data as well as provide meta data.  They are stored in the dumb data object as a HashMap.

Example:

ArrayList dataPoints = new ArrayList();
HashMap tags = new HashMap();
tags.put("format","phillips103");
tags.put("subjectId","NCC1701");
dataPoints.add(new IncomingDataPoint("ecg.V6.uv", 1420088400000, 5, tags));

Once your ArrayList contains all of the data points, it needs to be converted into a JSON array using GSON.

Gson gson = new Gson();
String json = gson.toJson(dataPoints);

Now all that remains is to open up the URL connection and send the data.  Since we're using the OpenTSDB HTTP API we'll be using port 4242 and the put operation.  An example URL string looks like this

String urlString = "http://myexampleopentsdb.com/api/put";

Next we open the url:

HttpURLConnection httpConnection = TimeSeriesUtility.openHTTPConnection(urlString);
OutputStreamWriter wr = new OutputStreamWriter(httpConnection.getOutputStream());

And we write our JSON array to it:

wr.write(json);
wr.flush();
wr.close();

We should also listen for the response code, since OpenTSDB will provide feedback that may be useful.

int HttpResult = httpConnection.getResponseCode();

Next, we talk about how to query the data back...