Gmail Oauth2 with Spring and JavaMail with Grails

Setting up Gmail Email Sending with Grails (2+, 3+) Service

Most Web applications will send some sort of email to users. Lost passwords, invitations to join. Its a good way to reach professional users. Not so great for young people (they never check them) but in order to send emails, you have to get the connection right. I didn';t build a particular groovy way, I used java code within a grails service to make a simple method available to controllers and other serivces:

interface EmailService {
    /**     
    * Send a set of emails and collect the results     
    * @param addresses list of addresses to send to
    * @param subject subject of email    
    * @param text  body of email   
    * @return a list of message IDs returned from the sending service    
    */    
    def send(List<String> addresses, String subject, String text)

    /**     
    * Setup any needed state from config files etc.     
    */    
    void initialize()
}

Most of the default grails java mail setup questions (I'm using spring mail with grails because its mostly already on the classpath) you will find involve turning off enhanced seucirty in your gmail account in order to get around the error message you will recieve on startup:

Authentication failed; nested exception is javax.mail.AuthenticationFailedException: 534-5.7.14 <https://accounts.google.com/signin/continue?sarp=1&scc=1&plt=AKgnsbvO
534-5.7.14 SC261Te39VZ5jtNBz2mvwNtIGtZLxYulCRb8D2u6rGTAg69U2-tQsPDzI1YPgWUbVo1ZQm
534-5.7.14 MlSxYEJHBzyTk-tQKy-6GN5HACShag4XcqNYlxbyWHYvMyMICSTPuwRFzM_Rn2kUOKLcoY
534-5.7.14 hcEmEC6i4DrvVh4h8KTTdK1VxgyDwD6QzfDxWgUa0vM7ZcLRfIURv1CThW4B0G5XvVgG3a
534-5.7.14 t-wJ_9P1uU8YoJK2c-QbrhZe2H9qo> Please log in via your web browser and
534-5.7.14 then try again.
534-5.7.14  Learn more at
534 5.7.14  https://support.google.com/mail/answer/78754 h24sm11972231ioi.17 - gsmtp

Someone just tried to sign in to your Google Account someone@gmailaddress.com from an app that doesn't meet modern security standards." - Oh great, here goes my day

This cryptic error results from using a setup like (in resources.groovy):

emailService(GmailEmailService) {
                template=ref("templateMessage")
                mailSender=ref("mailSender")
            }

templateMessage(SimpleMailMessage) {
                from="email@sendergmail.com"
            }

mailSender(JavaMailSenderImpl) {
                host="smtp.gmail.com"
                port=587
                protocol="smtp"
                username="email@sendergmail.com"
                password="yourpassword"
                javaMailProperties = [
                        "mail.transport.protocol" : "smtp",
                        "mail.smtp.auth" : true,
                        "mail.smtp.starttls.enable" : "true",
                        "mail.smtp.quitwait" : true,
                        "mail.debug" : true
                ]
            }

But what if you wanted to use enhanced security? Well available in the more recent versions of JavaMail is support of Oauth2.

To begin with though, you are going to need to follow the google guidelines here to recieve api credentials for your webserver. You should end up with a set of JSON credentials for your service account. :

{
  "type": "service_account",
  "project_id": "someid",
  "private_key_id": "SOMEID",
  "private_key": "YOURKEY",
  "client_email": "someemail",
  "client_id": "someid",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "someurl"
}

There is a branch here. You can choose to give the service account Delegate domain-wide authority (which I have done) or leave it at user-interactive mode. Since I dont want to have to ask my own company user account for permissions (which makes sense for a back-end webservice making sending email from only one user), I chose  to upgrade the account to domain-wide, which can be done in the console.

Adding Domain Wide Delegation means no prompts, but using an additional secret file

We will need to use these credentials to create oauth tokens that can be sent with each senmail request. To create these tokens (oauthtoken) you can use the Google Java API. (at the time of writing, the latest central maven available was: 'com.google.api-client:google-api-client:1.21.0') . Google has a guide available - but essentially you will be using the builder to setup a trusted account.

GoogleCredential credential = GoogleCredential.fromStream(new FileInputStream(authFile)).createScoped(Collections.singleton("https://www.googleapis.com/auth/gmail.send")) //for SMTP access only

Then email setup can be pretty easy. A lot of the boiler plate of creating a javax.mail.transport can be made easier by using some Java Code provided by Google itself.

If like us, you were using google apps accounts - there is additional information required. As you may get responses like:

DEBUG SMTP: SASL: no response
DEBUG SMTP: SASL authentication failed

Things to check:

* You have allowed the client id access to the same scope you are requesting (Google Scopes listing)
* You are creating your Google credential and access token with a user to impersonate:

GoogleCredential credential = GoogleCredential.fromStream(new FileInputStream(authFile))                .createScoped(Collections.singleton("https://www.googleapis.com/auth/gmail.send"))
credential.serviceAccountUser = "someuser@yourdomain.com" //important

Final Thoughts

After going through all of this trouble, I ended up switching to using the google api client and the gmail api client for java, while still using JavaMail for convience (google has a great guide here)

By the way, I reccomend using scheduling with retries in order to make sure your customers recieve their emails. It can be really annoying when an app fails to send that crucial forgot password email - they will often leave and not come back.

Turns out doing it the right way is a bit more involved - but worth it. One day your email's wont stop randomly working when google finally axes support for the basic authentication.

Unity3 and web services

Introduction
We wanted to be able to stream data into our Unity3D environment from our joomla website. I have rewritten the component from the ground-up twice but I thought I would share my design methodology (and technology) with the masses.

The humble backbone through all the iterations has been the WWW class (docs). By sending GET (and later POST) requests to the server, I would serve back information about articles, resources, images and other content. To being with, I wrote a couple extra methods in our joomla server's resource controller that send back the number of resources, the path to files. But this was all plain unformatted text, and one response per attribute - it was tiresome to use. I then moved to creating an XML structured response, and unity would then parse the XML. But because the XML readers required additional code, and I was trying to keep the codebase as lean as possible, I moved to JSON. A nice JSON reader comes prepackaged with the unity3 API for smartfox server - and I had already built a JSON web services API for a google web toolkit client... so the match was made in heaven - I was going to build a JSON web services api.

The result:

Picture! Images dynamically downloaded into Unity based on web services

So what follows is a big chunk of semi-source code of how that is integrated to Unity. The basic flow is:

• a UnityGameObject starts a coroutene asking for web data
• Generate request inside unity, and send using WWW class
• Server receives call, retrieves data from database, creates an encode a PHP object to JSON, sends this as response
• Unity receives object, decodes using libJSON, calls back original UnityGameObject with data

Example client functions for send request and decode response (C#):


    //Download and add image to billboard
    public static IEnumerator DownloadResource(int id, IHasResourceResult attachpoint)
    {


        WWW request = new WWW(baseURL() + hubCommandBaseUrl + "&task=jsonlist&jtask=detail&rid=" + id.ToString() + hubNoHtmlUrl);
//formats a url that look similar to:
//http://myserver/option=com_resource&task=jsonlist&jtask=detail&rid=1000&no_html=1
            yield return request;


            
            //attach resource callback
            //This is inside a loader help class, and called via a coroutene with a self reference passed as a callback
            //we need to callback the caller and give them the resource          attachpoint.AttachResource(loadFromJSON(request.text.Replace("[","").Replace("]","")));




            yield return null;
       
    }


//decode the JSON data

    private static ResourceResult loadFromJSON(String JSON)
    {
        LitJson.JsonData jResponse = LitJson.JsonMapper.ToObject(JSON);
        ResourceResult result = new ResourceResult();


        result.introtext = Escape((string)jResponse["introtext"]);
        result.id = (int) jResponse["id"];
        result.title = Escape((string)jResponse["title"]);
        result.thumbnail = Escape((string)jResponse["image"]);
        result.document = Escape((string)jResponse["document"]);


        return result;
    }



Example server functions to send response (PHP):

   //encode objects into JSON
function JSONObj($object)
{
$json =  json_encode($object);
$json = preg_replace( "/\"(\d+)\"/", '$1', $json );

return $json;
}
function startList()
{
echo '[ ';
}
function endList()
{
echo ' ]';
}

//called by controller to switch off to correct task
function JSONResponse($task)
{
switch ($task)
{
//switch off to task function
                        default:
                          $this->exampletaskfunction()
}
}

function exampletaskfunction()
{
$database =& JFactory::getDBO();
$id = JRequest::getInt( 'rid', 0 );
$resource = GET_RESOURCE_DATA_FROM_DATABASE()

                                       //make an object to convert into JSON
$obj = array();
$obj['introtext'] = htmlentities($resource->introtext);
$obj['image'] = '';
$obj['title'] = $resource->title;
$obj['id'] = $resource->id;
$obj['fulltext'] = $resource->fulltext? htmlentities($resource->fulltext): '' ;
$obj['document'] = '';


$this->startList();
echo $this->JSONObj($obj)."\n";
$this->endList();


}


Example JSON response data from server:

[ {"introtext":" \n\n\t The primary purpose of this paper is to present the instrumentation plan of a fullâ","image":"2011\/07\/03081\/.thumb\/.thumbfile.1.jpg","title":"SEISMIC RESPONSE OF STRUCTURAL PILE\u2010WHARF DECK CONNECTIONS FOR PORT STRUCTURES ","id":3080,"fulltext":"\t<p>\n\t <\/p>\n<p>\n\t The primary purpose of this paper is to present the instrumentation plan of a fullâ","document":"MjAxMS8wNy8wMzA4MS8ud2Vidmlldy8ud2Vidmlldy5zd2Y="}
 ]

Conclusion

So that shows a basic concept of how to program a web service architecture into unity.

Crossdomain policy file for Unity3 and SmartfoxServer

A little tip, now with Unity3, the webplayer will require a crossdomain XML policy for security (Info here). Make sure the xml file is utf-8 encoded, it saves a bunch of issues.


<?xml version="1.0"?>
<cross-domain-policy>
  <allow-access-from domain="*" />
</cross-domain-policy>


Make sure if you are using Smartfox Server (Product Website) (we use version 1.6 for our virtual classroom software) that you have auto send policy file to true. Without this, the socket connection will be block by the webplayer security model. Make sure you set the correct line in your config.xml and restart the server.


<ServerIP>*</ServerIP>
<ServerPort>9339</ServerPort>
<AutoSendPolicyFile>true</AutoSendPolicyFile>
<MaxUserIdleTime>120</MaxUserIdleTime>
<MaxSocketIdleTime>60</MaxSocketIdleTime>

Joomla error on authentication

So we just recently upgraded the database with fresh production data , but then the joomla site would not let us log on to either the front or back (admin) end. It would show a 500 error when you submitted the login form. We do use a LDAP authentication system, but surely that wasn't the problem?

In the error log the following line was repeated:

[Tue Jul 19 10:22:08 2011] [error] [client XXX.XX.XXX.XXX] PHP Fatal error:  Call to undefined method stdClass::onAuthenticate() in /www/neeshub/libraries/joomla/user/authentication.php on line 121

This was a call to onAuthenticate inside the (foreach $plugins as $plugin) block.

The fix: To actually get into the website and see what was going on, I needed to selectively disable our authentication plugins to find which one was failing. Of course I had to do this directly in the database, because I couldn't even get in to the plugin manager in joomla.