Uniqush Blog

if(we) blogs about Uniqush

2016-05-11T00:00:00+02:00

Misha Nasledov, Lead Platform Engineer at if(we), writes about using Uniqush at scale. if(we) sends tens of millions of push notifications every day using Uniqush. Check out her blog post! (Edited on 2018-05-15 to fix a dead link)

Release 2.1.0

2016-03-09T00:00:00+01:00

Please check out our release notes for version 2.1.0.

New APIs have been added: one for listing the subscriptions of a subscriber and another for showing the configurations of the push service providers for services that exist.

Concurrency issues in ADM and APNS have been fixed. The APNS implementation has been changed to use a new, reliable worker pool implementation.

This implementation has been running reliably at production scale, at a company named if(we), sending push notifications for the Tagged and hi5 social networks for a while now (nearly a year at time of writing). The company has 12 processes handling a peak of 60 APNS push notifications per second, and the tier is overprovisioned in order to be able to deal with traffic increases and potential surges. That's about 5 APNS push notifications per process per second. In addition, if(we) sends about 270 GCM push notifications per second distributed amongst these same processes, which adds about another 22 GCM push notifications per second each process sends. The subscribe API on the tier ends up getting hit about 500 times per second (about 42 queries per process per second). So, each process is handling about 70 queries a second easily. if(we) has been running without any issues for many months, especially after fixing some bugs that eventually manifested only at their scale.

A bug which could lead to infinite loops in rare circumstances has been fixed.

The APNS implementation no longer uses Go's default HTML escaping of JSON payloads. The APNS servers now render payloads with characters such as '"' properly.

More details have been added to error messages so that problems can be debugged more easily.

The APNS implementation has also been updated to deal with 100-byte APNS device tokens (previously 32-bytes) which Apple said it would roll out in 2016.

Release 2.0.0

2016-03-08T00:00:00+01:00

Please check out our release notes for version 2.0.0.

We've improved the response format of most APIs. Instead of sending server logs back, they now return JSON response objects. This makes it easy to distinguish successes from failures. For example, when sending push notifications to a subscriber with multiple delivery points, some may fail, but others may succeed. The JSON response object for a push contains easily parseable information on the status of each delivery point.

The APNS module has been updated to support the new 2048 byte (2kB) limit.

There are a few bugfixes, fixing both memory leaks and improperly closed connections.

This version officially removes C2DM support entirely, as C2DM was shut down by Google on October 2015.

The released binaries are built with Go 1.6. Compilation with Go 1.6 (or whatever the newest version may be) is recommended as the service will have lower CPU usage compared to builds using older versions of Go.

Release 1.5.2

2014-06-29T00:00:00+02:00

TL;DR: Please checkout our release note for version 1.5.2.

Sorry for my absence since November 2013. I was preparing other things and working on other projects. During this period, I received several bug reports related to uniqush-push. Some of them indicating some large service is currently using uniqush-push as their backend push service, while some of them contains very weird log messages which I did not expect. It is really exciting for me to see uniqush-push working on top of user base with millions of users. But it is so disappointed that I was not able to quickly found the root cause of all bugs.

Fortunately, it seems that most bugs are related to the APNS' waiting time calculation. When a push operation failed, uniqush-push will wait for some time before retry. It will announce a failure after 3 attempts. The waiting time is increased for each attempt. The way we used to calculate the waiting time was to square the previous waiting time. The problem is that we represent time period in terms of nanoseconds. Squaring it quickly leads to an integer overflow. This is why lots of logs in bug reports contains some negative waiting times.

There's also a potential problem with deterministic waiting times: It may leads to message storm. Imagine you want to send one million message at same time and some of them will fail. Because all of the failed messages will be resent after the same period, it leads to another spike of network usage, which may lead to failure as well. An easy fix is to wait for a random amount of time.

In 1.5.2, uniqush-push will randomly increase the waiting time before every resend of a message. This should fix most bug report I received. But I did not confirmed this with every reporter yet. If you have any other issues with the most recent version, please report a bug.

Release 1.5.1

2013-11-09T00:00:00+01:00

This version contains several minor bug fixes and improvements. Details can be found in the release note. Two noticeable changes are:

Subscriber and service names can contain @ now.
content-available parameter is supported in APNS.

Release 1.5.0: Kindle Support

2013-09-01T00:00:00+02:00

I'm very glad to announce that uniqush supports ADM since version 1.5.0, which means users can use it to push notifications to Amazon Kindle Tablets.

This is the only change in this version and since it changed the user interface, it is considered as a big change. (That's why we are no longer in version 1.4.x). As always, here's the release note.

I would like to share more details and plans in this post.

Kindle was a pain

When I first released uniqush, supporting Android and iOS is a very natural choice. They are popular and share the major market. I considered to support Windows phone soon, but because of lacking of device and environment (I use GNU/Linux only and do not have a copy of Visual Studio), this issue is now marked as "HelpWanted". It is wonderful if we support windows. But lack of windows phone support won't hurt us much.

Kindle, on the other hand, is an exception. It is an android device but Amazon chooses not to add GCM support. Moreover, it has a relatively large user group making it hard to be ignored. Amazon introduced ADM in Dec 2012 making yet another cloud service for developers to push notifications.

The release of ADM makes the idea behind uniqush-push more useful: Big companies create clouds with little mind of compatibility with others. For developers, a unified interface for push notification would be a very useful component. Interestingly, Amazon seems have same idea in mind. Just several weeks ago, Amazon released another service named SNS. What they are doing is quite similar to what uniqush-push is providing (and they charge users for it.)

Supporting ADM fixed the very piece of uniqush-push: Now, we can say that we supports major Android platforms with more confidence.

Chrome apps support is on the way.

Another issue on github asks for support of Chrome apps. I think it would be a very good step for the next release. (This means, we will quickly move to version 1.6.0 if we want to keep the same versioning convention.) I don't need to buy another device to support it (Buying a Kindle Fire HD cost me $159 even if it was on sale.)

Hey, do you remember uniqush-conn?

Yes, I do! And I would like to share some exciting news here: iOS client library is almost done. If you follow the uniqush organization, you may notice that we have a new member: nevo. I know him personally for years and he is a very competitive developer and a real expert in iOS development. Our iOS client library is developed by nevo and is working right now.

So what I'm waiting for? Why not release all the stuff? Well. First, you can always find the code on the uniqush organization. We never hind anything to the public. Secondly, releasing a software means taking responsibility. uniqush-conn would be an important component for our users' system. I have to make sure it works and works well.

Before releasing it, I would like to first write some app and actually eat my own dog food. An IM app will be developed to test the uniqush system as a whole. I will invite my family and friends to use the app. Once it was fully tested, I will release it along with detailed documentation.

Release 1.4.5

2013-08-01T00:00:00+02:00

This version contains only one bug fix. Since this bug is critical, it is highly recommended to upgrade to this version.

Thanks for cmabastar-gumi's bug report again. For more details, please check out our release note.

Release 1.4.4

2013-07-20T00:00:00+02:00

This version contains some minor bug fixes and a huge improvement for APNS. I would like to give special thanks to cmabastar-gumi for his detailed bug reports. Please check the release note for detailed ChangeLog.

In this post, I will explain more about how I improved the performance for APNS.

Overview

Both issue 24 and issue 27 described a poor performance when pushing notification through APNS. After a brief profiling, I made three changes:

Checking Feedback Service less frequently.
Using a connection pool to send notifications in parallel.
Response the request immediately after sending the notification, without waiting for the error response from APNS.

Checking feedback service less frequently

Starting from version 1.4.2, uniqush-push supports Feedback Service for APNS. Both version 1.4.2 and version 1.4.3 have a very naive approach for Feedback Service: uniqush-push checks feedbacks after every push operation. This is very inefficient because it has to establish a new TCP connection every time.

In version 1.4.4, uniqush-push checks the feedback for every 10 minutes. Although it may be better to provide a parameter for user to set this time interval to other values, I see no strong benefit by providing this parameter. At this point of time, I would rather use a constant time interval (10 minutes) and keep a simple interface for users.

Using connection pools

Before version 1.4.4, uniqush-push maintains one single connection with APNS for each APNS key/certificate pair. Clearly, using a connection pool could improve the performance when we have concurrent request, since we can send notifications in parallel through several TCP connections. However, APNS' document does not clearly say how many connections we can have for each key/certificate pair. According to discussion on stackoverflow, 15 connections would be find in most cases. In version 1.4.4, we use at most 13 connections for each key/certificate pair.

As a byproduct, a general connection pool library in Go is available on github.

No wait for the error response

OK. This is a trade off we made here. As an asynchronous service, APNS cannot guarantee the response time for the error response. What's worse, it does not even guarantee that there will be a response.

Fortunately, most errors can be checked before sending notifications to APNS. In version 1.4.4, uniqush-push returns success immediately once the notification has been sent to APNS. Some error like invalid token can be processed later inside uniqush-push. In most cases, we are good.

Conclusion

After making the changes above, the performance for APNS is improved ~100x. Let's quote cmabastar-gumi's test results mentioned in reply to issue 27:

we were able to send 22k messages in less than 3mins and we were also able to test around 68k users to be subscribed in around 5mins. Thanks for this build! It's damn fast!

Release 1.4.3 and uniqush-conn

2013-05-14T00:00:00+02:00

This version contains some minor bug fixes and includes some hidden features which will be used by uniqush-conn. The release note explains all the changes.

Instead of the changes themselves, in this post, I would like to talk about how they related to the uniqush-conn and what's the next step.

Hidden features? Why?

Well. This is a free and open source project. There is nothing hide from users. However, hidden feature here means something for internal use and may be changed or even deleted in the future. Users should not depend upon those features.

To make those features isolated from daily use, we introduced a set of parameters that users should avoid using them. When you send an HTTP request to uniqush-push, you send several parameters. Any parameters whose name starting from uniqush. should be considered as a reserved parameter and users should not use those parameters. This is a very easy way of separating name space and fortunately, uniqush is not a common word.

uniqush-conn? Are you still working on it?

Yes. Definitely. Good news: it is almost done and in fact, you can play with it right now. Moreover, you can even use our java library and learn how to use it with the example.

So what am I waiting for? Why not release it and have all the fun?

As I promised, there should be libraries for mobile clients. Right now, I'm still working on the android library and there is still zero line of code for iOS library. If I cannot support these two popular platforms with the same functionality, I should not release the code.

But you can still read the code if you are really impatient.

What's next?

In fact, my focus has switched to uniqush-conn and its related libraries in recent months. You can tell this from my github commit history, where I am extending my longest streak by working on these projects.

As I can expect, the android library will be coming soon. But the iOS library may experience some delay.

Release 1.4.2

2013-04-01T00:00:00+02:00

This version is mainly a bug-fix version. See the release note for more details. I will briefly explain each point below. (This version is released on April 1st. But it is really not a joke, I promise.)

Proper stop

bugfix Stop on /stop. Sorry, I forgot to implement the feature after refactoring the main package. Fixed issue 9.
bugfix Stop on SIGTERM. Similar with issue 9. User can now stop the program by sending SIGTERM to the process. Fixed issue 8.

The previous two bug fixes implemented what I promised in the document. Due to the refactoring in version 1.4.1, this feature was not implemented, accidentally.

Nil-pointer crash

bugfix Fixed a nil-pointer crash bug. The program will experience a nil-pointer crash (same thing as NULL pointer) under some conditions. For example, if user provides an invalid APIKEY for a GCM provider, then uniqush-push will crash when the user tries to push through this malformed provider. Fixed issue 10.

Thank you, Fabrizio! Your bug report helps me a lot on this bug.

Feedback Service in APNS

feature Support Feedback Service in APNS. Fixed issue 3.

I would say this is partly supported. The next version will support Feedback Service much better. In the Feedback Service's document, it says:

You should use the timestamp to determine if the application on the device re-registered with your service since the moment the device token was recorded on the feedback service. If it hasn’t, you should cease sending push notifications to the device.

However, uniqush-push does not keep the information like creation time of each delivery point. To avoid unnecessary unsubscription, uniqush-push will check the feedback after each push operation. This is not the optimal solution and it will be fixed in the next version.

1.4.1 and more

2013-02-25T00:00:00+01:00

Today, I have released uniqush-push 1.4.1, whose release note is available now along its binary files. It is highly recommended to update to this version, since there will be some non-deterministic bugs in old versions. In this blog, I will first go through the changes and explain some of them in details.

The changes in this version contains some improvements and bug fixes:

improvement On (un)subscription, the log message now contains service and subscriber information. Fixed issue 2.
improvement On adding an APNS push service provider, uniqush-push will report error immediately if the given certificate or key files do not exist. Fixed issue 6.
bugfix Removed all code related memory pool. There is no dependency on memory pool any more. Fixed issue 7.
improvement Rewrote the code in main package. It is more clean, simple and less redundant. Fixed issue 5.

I will explain them one by one below.

First two improvements

These two changes are not quite important but useful in its own right.

Apparently, adding more information into the log will help users and myself to identify the potential bug, which is generally a good thing as long as we do not leak sensitive information.

A system would be less error-prone if it reports error as soon as possible. Before this version, users can add an APNS push service provider even by providing a wrong path to the certificate and/or key files. The user can only know this error if they are actually using the malformed APNS provider to push something to a client. This is too late, and it should be better to report the error in advance.

No more memory pool

This is a hard decision to make.

I came up with the memory pool idea because of the famouse blog post. In the post, Russ Cox was able to optimize a Go program 15x faster than the original version, making it even faster than an equivalent C++ program. That post is amazing and I really like it.

In the blog post, Russ Cox said:

Having a garbage-collected language doesn't mean you can ignore memory allocation issues. In this case, a simple solution is to introduce a cache so that each call to FindLoops reuses the previous call's storage when possible.

Given the C/C++ background I had, the sentences above, to me, were like saying: use a memory pool under whatever condition if you want to have a fast program.

Then I made a memory pool. That is the beginning of the nightmare. I received several bug reports on the mailing list saying that they observed some very weird behavior. For example, some client received a message which should be sent to another one. I tried to re-produce their bugs bug failed.

Finally, I realized that the push service provider and delivery point structures were not cleared before putting back to the memory pool. This reminded me about the weird bugs on the mailing list.

I had two options: clear all data before putting them back to the pool; or drop the whole idea of memory pool.

Well. I do like a fast program, but I love simplicity as well. Managing my own memory using a garbage-collected language became a very awkward behavior to me. It is true that Go's garbage collector is not perfect right now. Even if it is good enough, it may need a good care of memory when writing programs anyway. But it is not my current concern. I want a simple program which can work. In fact, I even wrote a version with a correct behavior using memory pool. But I decided to not release it because of potential bugs caused by complexity.

Refactory

OK. this is a strong word for developers. Yes, I rewrote the whole main package.

I have to say that developing uniqush is really a learning process to me. I developed quite a few programs, some of them are even widely used. However, I haven't taken care of a project before uniqush. In uniqush, I have to write correct code while making sure it can evolve in a healthy way. This includes correcting my own mistakes and bad decisions in the code.

Refactoring the whole main package is correcting my bad design decision. I made a wrong choice of using channels and goroutines in Go making the main package cumbersome and hard to maintain. After this refactoring, the whole main package contains only four files and reduced around a thousand of lines of code. (You can see this fact from the code frequency.)

Besides, since I am going to support feedback service in APNS, such refactoring in inevitable. The service requires a new interface for push service providers and old main package can hardly support the new interface.

What about... uniqush-conn?

Sorry! I didn't expect that some essential parts are missing in Go and I have to write them by my own. More specifically, I have already written a Diffie-Hellman Key Exchange library in Go, and I am going to implement RSASSA-PSS signature algorithm later. Even though, I still want to use Go on server side, simply because it is too much fun.

With that being said, uniqush-conn will be delayed, again.

New release of uniqush-push: 1.4.0

2013-01-07T00:00:00+01:00

I am glad to announce the new release of uniqush-push: 1.4.0

Check out the release note for more details.

Changes in this version:

bugfix Fixed the bug on re-connecting APNS server. Thanks to Ivan Yu for his detailed bug report and analysis
feature Added loc-key, loc-args and action-loc-key support for APNS. You can send your localized message through uniqush-pus now. This part of code is adapted from benwei's patch. Thank you, benwei. Here is an example of using loc-key and loc-args: curl http://127.0.0.1:9898/push -d service=fake -d subscriber=client -d msg="hello" -d loc-key="lockey" -d loc-args="hello,world" Then the loc-args sent to the APNS will be ["hello", "world"]. Yes. it is comma separated. If one of your loc-key contains comma(,), then add a backslash (\) in front of the comma. (Similar to escape character in C)
feature Added support to true multicast for GCM.
feature Added time-to-live support for both GCM and APNS. It defines how long (in seconds) the message should be kept on push service provider's server if the device is offline.
change When subscribe to a GCM service, the user don't need to provide the account parameter. When unsubscribe a service, the user need to provide the same regid as the one he used to subscribe. For existing subscribers (i.e. the subscriber is added using previous releases), the account parameter is required when unsubscribe the service.
change This change only affects developers. The interface of PushServiceType is changed making it possible to support true multicast on GCM. Corresponding components are changed.

New version of uniqush-push

2012-11-19T00:00:00+01:00

We have just released a new version of uniqush-push, it's version 1.3.2 now. This version fixed some bugs and merged four repositories into single one. Details are in the release note. The two bugs fixed in this version were all reported by Ivan Yu which is one of uniqush's users and active on the mailing list. Special thank to him.

Making your own contribution to uniqush

2012-09-23T00:00:00+02:00

Uniqush now is receiving a growing attention from both users and developers. Some people have already contacted me personally or publicly saying they are willing to make there own contribution to uniqush. Some of them have already made a pull request on GitHub.

It is really an honor to receive those talent works, merge them into our repositories and make a better system through this process. To make it more efficient and less error-prone, a new document on how to make a contribution is online now. It is highly recommended to read this document first before making any contribution to uniqush.

Some highlighted part of the document:

For all source code repositories, there are three branches:

master branch, contains the most recent changes, which are tested and will be contained in the next release version. Normally, the code in master branch should compile and pass some basic test cases.

exp branch, contains the most recent changes, which may contain bugs, or even not be compiled. Every contribution should be made based on exp branch.

go1 branch (or release branch). If it is a repository contains go code, then this branch is named go1; otherwise, it is named release. As the name suggested, it contains the latest release version of the code. Please DO NOT make contribution based on this branch.

If you want to contribute your source code, please make sure your changes are based on exp branch, because we put our latest code there. It would be better to put your changed version in an individual branch or at least in exp branch, and then send your pull request on github. Before doing any work, you may want to read this document from github.

Please feel free to contribute your code, bug reports, ideas, suggests to this project. Whatever your contribution is, it is essential to us and all users in this community. We appreciate your support!

New Feature: push with wildcard

2012-08-10T00:00:00+02:00

NOTE: This feature is included in version 1.3.1 and later. Please check out our latest release from the download page.

I would like to announce a new feature in uniqush-push: wildcard. More precisely, the asterisk sign (*).

Just like wildcard in UNIX system, you can use asterisk sign (*) to match zero or more characters to specify more than one subscribers.

Here is an example (again, I use curl to demonstrate. Of course, you can use any language with HTTP library to communicate with uniqush-push):

curl http://127.0.0.1:9898/push -d service=myservice -d subscriber=uniqush.* -d msg=”Hello World”

This command will push a notification to any subscriber whose name starts with "uniqush."

And of course, if you like, you can use this feature to push message to all subscribers:

curl http://127.0.0.1:9898/push -d service=myservice -d subscriber=* -d msg=”Hello World”

This is a very useful feature if you want to classify your subscribers in groups. For example, I have two groups, namely group1 and group2. We can have a naming convention, say all subscribers under group1 should have names start with "group1.". If we want to send a message to all subscribers in group1, we can simply say, send it to group1.*. It's very easy and convenient.

One side effect brought by this feature is a more strict naming rule for services and subscribers. Any service/subscriber's name should only contain: 26 English letters (a-z and A-Z), numbers(0-9), dash (-), underscore (_), dot (.). In short, it should match this regular expression: ^[a-zA-Z0-9-\_\\.]+$. Any other character should not be used in service/subscriber's name.

NOTE: Asterisk sign (*) is the only wildcard we support right now. I see no reason to support more wildcards at this time (like ?, [], etc.). If you have any comment/suggestion, it would be more than welcome.

Unqiush.org is upgraded

2012-08-04T00:00:00+02:00

As we mentioned in the previous update, the website of uniqush.org is now upgraded. All content in previous pages are updated and the whole website is based on static pages generated by webgen. Besides, we provide binary packages for 64-bit systems, including .rpm, .deb and .tar.gz.

Anohter news is: the uniqush organization is working now to replace the old uniqush repository under my github account. This change will not affect any function of uniqush-push, but will help us to maintain a more organized code base and provide an easy way to contribute.

By the end of August, we will release version 1.3.1, which will provide the following functions:

uniqush-push will support wildcast when you push some notification to subscribers. For example, you can name your subscriber as group1.sub1, group1.sub2, group1.sub3, etc. Then you can say "push message to group1.*" and the message will be push to all subscribers whose name starting from group1..
uniqush-push will support true multicast. Because [GCM] introduced multicast, we need to support it in native way.

Big changes are coming soon

2012-07-31T00:00:00+02:00

Given the plan in uniqush after go1, some changes, which may be considered as big changes, will be made in the coming August.

A new static website for uniqush.org. Yes, no more trac.
A new uniqush organization on github is created. All uniqush related code will be moved under uniqush organization. It is more than welcome to join it.
uniqush will provide binary packages for 64-bit GNU/Linux systems, including .deb package, .rpm package and .tar.gz package (which can be used on all 64-bit GNU/Linux systems). It is highly recommended to install uniqush system from those packages. No more go get (but you can still use go get anyway if you like).
Last but not least, the release of uniqush-conn and other stuff will be postponed (sorry about that). Hopefully, I could make it before Feb. 2013. Watch the repositories on uniqush organization for more updates.

Here are some details about these updates.

Home page will be static

Check out our uniqush-www repository, the whole website will be there. It is a static website created by webgen.

Reason:

I found some spam using uniqush's ticket even if I have already added CAPTCHA plugin, which is very annoying.
I don't quite like to backup databases for wiki pages on uniqush, although it is done automatically by two cron jobs running on uniqush.org and a small Arm server in my home. Recovery the whole website from scratch is still hard even if I have all content on my hand (import data to database, setup trac and its plugins again, etc.)
If I use webgen based static pages, I can easily change the look of the website by changing the template.

The website will be replaced once I migrated all pages on uniqush.org. So please expecting the outage of uniqush.org for one or two days. In the meantime, you can still access our source code from uniqush organization.

Uniqush organization on github

The uniqush organization is meant to cover most development-related operations on uniqush system --- including uniqush-conn, uniqush-push, uniqush-client and their issue tracks. This job could be done better by using an individual organization account, rather than a personal account. Moreover, considering there are already some people contacted me saying they are interested to join the uniqush system development team, an organization is the best fit for such team work in the future.

However, because we will migrate source code to uniqush organization, programs, if any, using code from uniqush-push should be aware of the changed url.

Binary packages

One of the beautiful parts of Go programming language is: it deliberately does not support shared library or dynamic library. That means the program written in Go programming language is self-contained, i.e. one executable, no dependencies (in the sense of library, not external program). Once the program is produced from compiler, you can copy it around and it can be always run regardless what ever the libraries you have installed on the machine.

This makes me very easy to produce binary packages, which should be a necessary part for an free/open-source project.

Right now, all packages are produced and will be on-line after replacing the website with static pages.

uniqush-conn, and other stuff

I have to appologize for the delay. I over-estimated my ability and under-estimated the workload I am facing right now. However, the good thing is: I have a more concrete design on the whole system.

uniqush-conn, as always, will be written in Go programming language. The core part of uniqush-client will be written in C for portability and performance consideration. There will be several derived work on uniqush-client to support different platforms including Android, iOS. This means the core part of mobile library is written in native C language, which, as you can expect, will be and should be fast.

Misc

uniqush-push will support wildcast when you push some notification to subscribers. For example, you can name your subscriber as group1.sub1, group1.sub2, group1.sub3, etc. Then you can say "push message to group1.*" and the message will be push to all subscribers whose name starting from group1..
uniqush-push will support true multicast. Because GCM introduced multicast, we need to support it in native way.

All features above will be included in the next release, verion 1.3.1

GCM is supported now

2012-06-29T00:00:00+02:00

Google announced GCM on Google IO 2012 two days before. It is a push notification service to replace C2DM.

Two days later, you can use uniqush to send notifications to android devices using GCM. The latest API document contains details on using GCM.

The latest version of uniqush now is 1.3.x. Instead of installing from source code, binary packages including .deb and .tar.gz format will be online soon. (No later than July 2, 2012.)

Recent update on uniqush-push

2012-05-08T00:00:00+02:00

As I mentioned in uniqush after Go 1, uniqushd is now renamed to uniqush-push. However, it is more than a renaming work. I will review some changes on uniqush-push in this blog post.

HTTP responds more information

Users communicated with uniqushd through HTTP. Whenever a request was received, uniqushd would reply an HTTP response containing only one line: id=xxxxx. It is the request ID which helps users check the status in log. There was no other information responds from uniqushd. Users have to check log to see if a push was successfully sent to the cloud.

uniqush-push now uses HTTP as communicating protocol as well, but provide more information inside the HTTP response. On adding push service provider or subscribing a service, it responds the service provider's ID or delivery point's ID respectively. On push, it responds Success! on, well..., success. On failure, it responds the reason. But there is one exception: on retry, which usually happens on some temporary failure, it only respond Retry and users have to check log to see if the message is successfully delivered.

Fixed a bug

Thank @M_Cainelli for his bug report.

Description of this bug: send subscribe command to uniqush-push several times with same delivery point under same subscriber, push a message to the subscriber, there will be multiple messages pushed to the delivery point.

This is because of the cache mechanism in front of database. On adding a delivery point to a subscriber in the cache, we forget to check if the delivery point is already under the subscriber. This will then cause the multiple time of delivery.

Future

Besides, we have re-arranged the namespace preparing the comming of uniqush-conn. I am busy on designing the protocol between uniqush-conn and client. Hopefully, we could release uniqush-conn on time.

Uniqush after Go 1

2012-03-28T00:00:00+02:00

Today, March 28th 2012, is a big day for every gopher: Go team released Go version 1. This means Go -- both the language itself and its standard library -- is stable now. Because of this, I think it's time to move uniqush onto next stage. I would like to share the blueprint of uniqush in this article.

Instead of simply providing push service, uniqush will try to provide communication facilities for mobile developers, including both server-side developers and mobile-side developers. This change will make the current work of uniqush, which only has one single program -- uniqushd, become a (relatively small) part of the whole uniqush project.

In the future, the uniqush project will mainly consists of four parts: uniqush-push, uniqush-conn, uniqush-front and uniqush-mobile-libs. I would explain each of them in the following sections.

uniqushd becomes uniqush-push (done)

Currently, uniqushd is the only program we provide in uniqush project. The function of this program is to behave as a proxy between third party server and push service cloud (C2DM, APNs, etc.) so that the third party server can push messages to many platforms (Android, iOS) in a consistent syntax without knowing protocols on different clouds.

In the future, uniqushd will be renamed to uniqush-push because it actually only deals with push service. It will support more platforms (Windows Phone, Black Berry, etc.), but there is no more functions added to this program. It only focuses on one single task: push a message through different clouds.

uniqush-conn handles connections between server and apps

When I say server, I mean an entity that may consists of more than one node. This means it may be a cloud, a cluster, more precisely, or even a datacenter. (OK. it may be, if they want, several datacenters.)

Imagine you have an app, let's say an online chat app, needs to connect your server directly once it is active. You may define the word active under your context. In most cases, it is when the app is in front, i.e. user is using the app on his phone now. If the server sends a message to the app now, the server may want to use this direct connection instead of push service cloud to send this message.

In an example of online chat app, user A sends a message to user B. User A first start his app on the phone. The app then connects to the server saying "I'm A and am online now". From now on, the app on user A's phone is active and a connection between the app and the server is established. User A wants to chat with B. He types in some message, click send button and the message is sent to the server through the newly established connection. The server unpacks the message seeing it is a message for B. It checks whether B is online/active now. If B is online, then there's a connection between the server and B, the server can send the message through this connection. If B is not, then the server will send some message, saying "Hey, B, you got a message from A, check it!" through push service clouds (C2DM, APNs, etc.).

If the server in the previous example is using facilities from uniqush, then it may use uniqush-push to send message through push service clouds, and use uniqush-conn to deal with active connections between the server and apps.

In fact, the server does not even need to know if a certain user is online or not. uniqush-conn can deal with it and decide whether send message through a direct connection, or through uniqush-push.

Now we can see the benefit. As a server-side developer, you only need to send a message to uniqush-conn, and say "Send this message to user B". Then you are done. What if the user B was offline? Well, uniqush-conn could then automatically tell uniqush-push to push a notification to user B. If the message if too big to be fit into a notification, uniqush-conn will send a notification to user B, saying ``You got a message'', keep a temporary copy of that message and send the message to user B once it's online. The server even does not know the existence of uniqush-push.

uniqush-front handles scale-out growth

As I mentioned, server side may consists of more than one node. With the increasing of number of connections, growing in a scale-out way should be a concern for server developers. We introduce uniqush-front to handle connections distributed among several instances of uniqush-conn in more than one nodes (or cloud instances). The server-side program will finally only communicate with uniqush-front without knowing which instance of uniqush-conn handles which connection. The protocol between server and uniqush-front will be exactly the same as the protocol between server and uniqush-conn. So that changing from uniqush-conn to uniqush-front will not lead to change of code.

Suppose the server side has 10 nodes. Each node runs an instance of uniqush-conn. A certain user may connect one of the instance on some node. The server side program only need to tell uniqush-front what message needs to be sent to which user. Then uniqush-front will figure out which instance of uniqush-conn should be use. Adding more node will not be a pain because the server side program won't be changed.

uniqush-mobile-libs helps app developers

uniqush-conn keeps connections between apps and the server. It also sends messages, commands to apps. The app developers do not need to know details about how to communicate with uniqush-conn. uniqush-mobile-libs will contain libraries for popular mobile platforms so that app developers can use it to communicate with uniqush-conn, hence with the server side program.

When?

uniqush-push is already there. I only need to change its name from uniqushd to uniqush-push. I will start developing uniqush-conn in this week and (hopefully) uniqush-conn and uniqush-mobile-libs will be released before Sep. 2012.

I wish I can make it faster. But sorry for disappointing you, it seems impossible for me: there is only one developer in this project, and yes, I am the developer. If you are interested in this project and want to make your own contribution, please feel free to contact me.

Update

As I mentioned, uniqush-push is online now. It is more than a renaming work. All related documents are changed respectively. If you find any inconsistence between the document and the code, please let me know. Thank you very much.

Hello World! Hello Uniqush!

2012-03-02T00:00:00+01:00

I started uniqush in Aug. 2011, when I was reading documents about push service. The idea of this project is simple: build a software as a proxy between the app server and push services, so that the server side programmer could use a simple and consistent interface to push message to any mobile device.

After about one month of development, the first functional version was online and also the web site. Once it becomes a free/open-source software project, it is more than coding work. Maintaining such a project is not a trivial work even for experienced developers. Although I did not widely announce the birth of uniqush, I did make my best effort to build a bridge between my project and my (potential) users. That is why I spend lots of time on drawing work, and hence this blog.

Interaction with users are crucial to a software project. Although github provides plenty of means for such purpose, a blog may be a simple and effective way to make announcement and get feedback immediately.

Hence, I set up this blog hoping to hear the voice from the early or potential users of uniqush. If you have any comment on this project, please let me know.