Code-modifications for RT at UiO

To adapt RT for use at the University of Oslo, we had to modify the source and add some local overrides to get it to integrate properly into our existing systems. This document explains and documents these modifications.

This document is a technical summary of the changes for use when upgrading to a newer version of RT, or for use by others needing to duplicate the modifications.

Information on how to modify RT using the available hooks is available from the RT wiki (wiki.bestpractical.com/index.cgi?CleanlyCustomizeRT).

LDAP-synchronizing and authentication

We want to check passwords using LDAP when logging into the RT web page for users in the UiO LDAP database (internal users), and check the password using info in the RT database for external (non-UiO) users. LDAP should be authorative for internal users, so password changes for internal users should be blocked in RT.

We want internal users to have the same username as they have in the UiO system, instead of the email address. This means emails received from internal users must map to the correct user name.

When a email from a unknown (to RT) user is received, we want to look up the email address used using LDAP, and get the user name associated with this address. The user name is then used to create this user in RT, and the a new ticket is created or the message is appended to an existing ticket. The email response should be sent to the official email address of the user in question.

People with several user names at UiO will get several users in RT. External users will get an RT user with user name set to the email address used.

We had to modify the access lists in the LDAP server to allow access to the email LDAP tree from the RT servers. I also had to install Net::LDAP on the machine in question.

The design and implementation of this feature is based on code from Stewart James (thefeed.no/marcus/rt/User_Local.ldap.pm.txt) and Carl Makin (www.gossamer-threads.com/lists/rt/users/35664). It gave me a starting point and clues on the hooks to use. Some approaches presented at RRWF on sourceforge (cvs.sourceforge.net/viewcvs.py/rrfw/otherstuff/rt_ldap_integration.html?rev=1.3) where considered and rejected, as the apache ldap authentication would block email submissions and make it impossible for external users to authenticate.

LDAP sync and auth is implemented by adding hooks in RT::User::CanonicalizeUserInfo() and RT::Interface::Email::Auth::MailFrom::GetCurrentUser(), making sure they call RT::User::LookupExternalUserInfo() based on the email address and real name given in emails. This patch is sent upstream to rt-devel 2004-08-29. These changes are in local/lib/RT/User_Local.pm and local/lib/RT/Interface/Email/Auth/MailFrom_Local.pm.

Next, the functions RT::User::IsPassword(), User::SetPassword() and RT::User::LookupExternalUserInfo() are overrided to use LDAP information for internal users, and the normal RT user database for external users. These changes are in local/lib/RT/User_Local.pm. The hooks to make it easier to override IsPassword() and SetPassword() is sent upstream to rt-devel 2004-08-31.

The local overrides are placed in local/lib/RT/User_Local.pm and local/lib/RT/Interface/Email/Auth/MailFrom_Local.pm, and this configuration is placed in etc/RT_SiteConfig.pm:

# LDAP configuration for User_Local.pm
Set($LDAPExternalAuth, 1); # Enable LDAP auth
Set($LdapServer, "ldap.uio.no");
Set($LdapCAFile, "/site/w3-sertifikater/w3_cacert.pem");
Set($LdapUser, ""); # Can search without username and password
Set($LdapPass, "");
Set($LdapAuthStartTLS, 1); # Need to use TLS or ldaps to check passwords
Set($LdapAuthBase, "ou=users,dc=uio,dc=no");
Set($LdapAuthUidAttr, "uid");
Set($LdapAuthFilter, "(objectclass=posixAccount)");

Set($LdapMailBase,       "ou=mail,dc=uio,dc=no");
Set($LdapMailFilter,     "(&(objectClass=mailAddr)(targetType=user))");
Set($LdapMailScope,      "one");
Set($LdapMailSearchAttr, "mail");
%RT::LdapMailResultMap =
    (
# From LDAP subtree ou=mail,dc=uio,dc-no
     'target'             => 'Name',
     'defaultMailAddress' => 'EmailAddress',
# From LDAP subtree ou=people,dc=uio,dc-no (not yet implemented)
#     'cn'            => 'RealName',
# Same tree, only employees
#     'uid'             => 'ExternalAuthId' (?)
#     'street'          => 'Address1',
#     'postalAddress'   => 'Address2',
#     'telephoneNumber' => 'WorkPhone',
#     '?' => 'City', # Not present (part of 'street')
#     '?' => 'Country', # Not present
#     '?' => 'MobilePhone', # Not present
     );

In addition, a script rt-sync-ldap was written to keep the information in RT in sync with the information in LDAP.

Files:

Email-administration of RT tickets

We want to be able to update ticket attributes using email, to enable for example owners updates and queue changes of a ticket using email. This is done using pseudo-headers at the top of the emails sent to RT. These pseudo-headers can be used both on ticket creation, and when adding information on a ticket. Only the first paragraph in an email is checked for pseudo-headers.

The following table show the fields recognized by the pseudo-header parser. Fields marked with a '+' can have multiple values, by adding several such fields to one email. Fields enclosed with () are parsed but not yet handled.

Attribute Value
Subject Text string with new title
Queue Name of new queue for this ticket
Status new, open, stalled, resolved, rejected or deleted
Due Date. RT understands both seconds since epoch and several text formats like the ISO 8601. RT uses Date::Manip to parse dates.
Starts Date
Started Date
Owner RT username
+AddRequestor Email address
+DelRequestor Email address
+AddCc Email address
+DelCc Email address
+AddAdminCc Email address
+DelAdminCc Email address
TimeWorked number (minutes)
TimeEstimated number (minutes)
TimeLeft number (minutes)
Priority Priority value (0-99)
FinalPriority Priority value
(+DependsOn) Ticket id #
(+DependedOnBy) Ticket id #
(+RefersTo) Ticket id #
(+ReferredToBy) Ticket id #

Here is an example email demonstrating several of the attributes modifiable using email:

From: Mari Wang 

The implementation is based on a patch from Matthew J. Draper posted to rt-devel. The patch had some shortcomings, but contained useful ideas on how to get this working. The implementation was done by Mari Wang.

Three changes were needed to get this working. First, users needed to be granted the right to modify tickets using email. This was done with a trivial email plugin named Auth::GiveRights, granting this right to everyone. Next, the email parser (Email::Gateway) needed to include a hook to request ticket status updates based on the content of the incoming email. Finally, several methods in RT::Ticket concerning updates of ticket info based on email content needed some updates and corrections.

Authorization is done by trusting the From: email header, looking up the RT user for this email address, and granting rights to modify ticket attributes if that user has these rights in RT. An alternative is to use the Auth::GnuPG email plugin to do this, and requiring that all users wanting to update ticket attributes using email need to sign their email with GnuPG. Trusting the From:-header was decided to be good enough for the testing period. One might want to reconsider this later.

The hook in RT::Interface::Email::Gateway() is needed to make sure both new tickets and existing tickets can be modified using email. If only existing tickets were to be modifiable, one could use the existing MailPlugin hook and call the code to parse the ticket for pseudo-headers from there.

The parsing of pseudo-headers is done using RT::Ticket::UpdateFrom822(). The function was present in RT already, but needed some updates and corrections to work properly. The code to handle custom fields (_UpdateCustomFieldsFrom822()) is not enabled, as it isn't tested.

To enable the authorization code, etc/RT_SiteConfig.pm is updated with the following configuration options:

# Configure Email attribute updates
push(@RT::MailPlugins, "Auth::GiveRights"); # put it early in the list

Files:

  • local/lib/RT/Ticket_Local.pm (diff, copy)
  • local/lib/RT/Interface/Email_Local.pm (diff, copy)
  • local/lib/RT/Interface/Email/Auth/GiveRights.pm (copy)

 

In 2006 we got BP to integrate the email administration system into RT proper, and fix a few of the problems we had with the current system. It is now available as RT-Extension-CommandByMail from CPAN.

Automatic password generation for new external users

 

We want new external users (non-LDAP users) to receive their RT password when they submit a message into RT for the first time, to make sure these users can log in and check the status of their tickets in RT.

 

This is done by modifying a RT template. This approach (wiki.bestpractical.com/index.cgi?AutogeneratedPassword) is described in the RT wiki, where I found it.

 

Visit 'Configuration->Global->Templates' and select the 'Autoreply' template. Rewrite it to look like this (the active part is the perl script fragment in the middle):

Subject: AutoReply: {$Ticket->Subject}


Greetings,

This message has been automatically generated in response to the
creation of a trouble ticket regarding:

"{$Ticket->Subject()}",

a summary of which appears below.

There is no need to reply to this message right now. Your ticket has been
assigned an ID of [{$rtname} #{$Ticket->id()}].

Please include the string:

[{$rtname} #{$Ticket->id}]

in the subject line of all future correspondence about this issue. To do so,
you may reply to this message.

{
*RT::User::GenerateRandomNextChar = \&RT::User::_GenerateRandomNextChar;
my $user = $Transaction->CreatorObj;
if (($user->id != $RT::Nobody->id) &&
(!$user->Privileged) &&
($user->__Value('Password') eq '*NO-PASSWORD*') &&
($user->Name =~ m/\@/) ) {
my ($stat, $pass) = $user->SetRandomPassword();

if (!$stat) {
$OUT .=
"An internal error has occurred. RT was not able to set a password for you.
Please contact your local RT administrator for assistance.";
}

$OUT .= "
You can check the current status and history of your requests at:

".$RT::WebURL."

When prompted, enter the following username and password:

Username: ".$user->Name."
Password: ".$pass."

";
}
}
Thank you,
{$Ticket->QueueObj->CorrespondAddress()}

-------------------------------------------------------------------------
{$Transaction->Content()}

 

This approach make sure external users only get one reply email (as opposed to the scrip-based method). Both methods leave the assigned password in the text associated with the ticket, so the user should be encouraged to change the password as soon as possible.

 

For this to be useful, one need to give new users the right to see their own tickets. I did this by granting the right 'ShowTicket' to the system group Everyone. Visit 'Configuration->Global->Group Rights' and select 'ShowTicket' for the system group Everyone. Press 'Modify Group Rights' to activate this change.

 

It might be useful to change the autoreply to include the URL to the ticket info page https://rt-dev.uio.no/Ticket/Display.html?id=#.

 

Spam-handling

 

We want to sort suspected spam messages into a special spam queue, and assign the task of checking this queue to some individual reassigning real messages into the proper queue. It would be nice to be able to throw away highly suspicious email, and only have to manually classify the emails with less clear spam status. We let the UiO email system do the classification (using spam assassin and other mechanism, and look at the X-UiO-Spam-Score header to decide if the email look like spam or not.

 

This is done by implementing a RT MailPlugin Filter/SpamHeader.pm, and let this filter look at the configurable header and either drop the emails or put them in the configurable queue. The filter is placed in local/lib/RT/Interface/Email/Filter/SpamHeader.pm, and etc/RT_SiteConfig.pm is updated with the following configuration options:

# Configure Spam detection
Set($SpamHeader, "X-UiO-Spam-Score");
Set($SpamLowMatch, "ss+");
Set($SpamLowQueue, "spam-suspects");
Set($SpamHighMatch, "ssssss+");
Set($SpamHighQueue, undef); # throw away
unshift(@RT::MailPlugins, "Filter::SpamHeader"); # put it first in the list

 

The code to implement this is sent upstream to rt-devel 2004-08-31.

 

Files:

 

  • local/lib/RT/Interface/Email/Filter/SpamHeader.pm (diff)

 

Automatic ticket priority escalation

 

We want tickets to increase their priority over time, to increase the chance of someone fixing them before they get too old. our solution is based on the description available from the RT wiki on Escalation configuration (wiki.bestpractical.com/index.cgi?ConfigureEscalation).

 

I created wrapper script (cron-escalate) to loop over all queues, and called this from cron every night, as described on the wiki escalation script example (wiki.bestpractical.com/index.cgi?ConfigureEscalationExample). To get this script running, I had to make sure the unix users running the cron job also existed as an RT user, and that this RT uses had the global privileges ShowTicket and ModifyTicket. I decided to use the rt-user unix user for this task, and created the RT user with the appropriate rights.

 

To get the cron jobs running, the rt-user unix user needed write access to the RT log file var/log/rt.log. It need to be writable by the apache user (nobody) as well, so I made sure both nobody and rt-user had read/write access to this file by leaving it owned by nobody, and changing the group to uio-rt (one of the groups of rt-user), and giving the group write access to this file.

 

This is the content of local/bin/cron-escalate:

#!/site/perl-5.8.4/bin/perl
#
# Author: Petter Reinholdtsen
# Date: 2004-08-31
#
# Based on shell version previously on
# <URL:http://wiki.bestpractical.com/index.cgi?ConfigureEscalationExample>
# and
# <URL:http://wiki.bestpractical.com/index.cgi?ConfigureEscalation>
#
# Run from cron as user rt-user, and make sure rt-user have
# permissions ShowTicket and ModifyTicket

use strict;
use warnings;

# Location of RT3's libs and scripts
use lib ("/site/rt3/lib", "/site/rt3/local/lib");
my $crontool = "/site/rt3/bin/rt-crontool";

package RT;
use RT::Interface::CLI qw( CleanEnv );

# Clean our the environment
CleanEnv();

# Load the RT configuration
RT::LoadConfig();

# Initialise RT
RT::Init();

my $queues = new RT::Queues($RT::SystemUser);
$queues->LimitToEnabled();

# escalate tickets for all queues
while (my $queue = $queues->Next) {
my $queuename = $queue->Name;

system("$crontool --search RT::Search::ActiveTicketsInQueue " .
"--search-arg \"$queuename\" ".
"--action RT::Action::EscalatePriority");
}

 

This is the cron job configured for the rt-user unix user:

# Escalate tickets once a day, 04:25. [pere 2004-008-31]
25 4 * * * /site/rt3/local/bin/cron-escalate

 

Files:

 

  • local/bin/cron-escalate

 

In 2006 we got BP to write us a custom escalator system to fix a few of the problems we had with the default system. It is available as RT-Action-LinearEscalate from CPAN.

Automatic reminders to ticket owners

 

We want to send out automatic reminders to ticket owners, to make sure no tickets are forgotten. I found some configuration examples for this from the contributions at the RT wiki (wiki.bestpractical.com/index.cgi?Contributions), and later found a newer version of the rt-remind script available from the authors site.

 

The script was extended to loop over all enabled queues (patch sent upstream), and the configuration was modified to use local email addresses. There are several configuration options we need to figure out:

 

  • Mail address to use as the from address.
  • Mail address to notify for tickets owned by Nobody and root. This can be configured per queue. It might be a good idea to send these to the queue admin CC watchers instead of configuring it in the script.
  • Priority/state limits when sending out reminders.

 

The script is intended to send out reports every day, week and month, with new and unassigned tickets sent out every day, and a summary of more tickets are sent out every week and month. A wrapper script cron-remind was created to make sure only one email is sent per day. This is the crontab set up for unix-user rt-user to make this happen:

# Notify ticket owners, daily, weekly and monthly
10 7 * * * /site/rt3/local/bin/cron-remind

 

Files:

 

  • local/bin/rt-remind
  • local/bin/cron-remind (diff)

 

Use References and In-Reply-to of incoming emails to find ticket ID

 

We want RT to understand the mail headers References and In-Reply-to, to make sure it will insert new emails into the correct ticket even if the ticket markers are missing in the subject of the message.

 

To get this to work, we needed to get RT::Attachment to insert the message id into the table column intended for such use, and the email reciever to use this information. A new method RT::Ticket::LoadByMessageId() was added to support the mail parsing. These patches are sent upstream to rt-devel 2004-09-10.

 

Files:

 

  • local/lib/RT/Attachment_Local.pm
  • local/lib/RT/Ticket_Local.pm
  • local/lib/RT/Interface/Email_Local.pm
  • Complete diff.

 

Use ISO-8601 format as display format

 

We want to use ISO 8601 format when displaying dates. We do not want to use the date format used in USA.

 

This was implemented by adding a replacement AsString method to the RT::Date class, and placing it in Date_Local.pm

 

Files:

 

 

Drop bouncing addresses as ticket watchers

 

We send a lot of different mailing lists into RT, and some of them are not detected by the regex in RT::RTAddressRegexp, so we want to remove them as watchers to avoid lots of bounces. Wrote a script to run daily to remove all queue addresses from tickets.

Files:

 

 

Provide password reminder form on the front page

 

External users need to be able to get a new password if they forgot about the old one, without contacting the support staff. We made a small modification to the front page to make this possible. Place the file in /local/html/Callbacks/somename/Elements/Login/AfterForm to enable this feature.

Files:

 

 

Planned updates

 


 

Published June 25, 2010 10:49 AM - Last modified July 6, 2010 1:00 PM