Monday, February 13, 2012

Compiling Ruby MRI on Mac OS X

EDIT: include libyaml

Mac OS Snow Leopard comes with a pretty old Ruby:

$ ruby --version
ruby 1.8.7 (2010-01-10 patchlevel 249) [universal-darwin11.0]

You can download and compile the latest Ruby from source. You'll need to have XCode and libyaml installed first. Download libyaml from http://pyyaml.org/wiki/LibYAML and extract it in your Source directory:

$ cd Source
$ tar xzf ~/Downloads/yaml-0.1.4.tar.gz
$ cd yaml-0.1.4/
$ CC=clang ./configure --prefix=/usr/local/yaml-0.1.4
... output omitted ...
$ make
... output omitted ...
$ sudo make install
... output omitted ...

Download the latest Ruby from http://www.ruby-lang.org/en/downloads/ and then extract it in your Source directory:

$ cd Source/
$ tar xzf ~/Downloads/ruby-1.9.3-p0.tar.gz
$ cd ruby-1.9.3-p0/

Configure for compilation with clang and installation in /usr/local/:

$ CC=clang LDFLAGS=-L/usr/local/yaml-0.1.4/lib CPPFLAGS=-I/usr/local/yaml-0.1.4/include ./configure --prefix=/usr/local/ruby-1.9.3-p0
... output omitted ...

Compile:

$ make
... output omitted ...
$ make test
... output omitted ...
PASS all 943 tests
... output omitted ...
PASS all 1 tests

(I tried that with the latest stable, ruby-1.9.2-p290, and some tests failed there, so I didn't use it.)

Install:

$ sudo make install
... output omitted ...
$ /usr/local/ruby-1.9.3-p0/bin/ruby --version
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin11.2.0]

You can either always specify the path to it or you can add it to your PATH variable.

Monday, February 14, 2011

Making an executable Ruby archive distribution

As a follow-up to my previous post, you can even make a single executable file of your Ruby archive. It relies on the fact that zip files are pretty resilient to extra garbage data in the beginning. Let's say I have two files, lib/greetings.rb and bin/hello.rb. greetings.rb defines a make_hello() function that is used by hello.rb. Here's hello.rb:

require "lib/greetings"

puts "Enter your name:"
name = gets.strip

puts "\n" + make_hello(name)

You can run it like this:

$ ruby bin/hello.rb 
Enter your name:
Steve

Hello, Steve!
$

First, we make a zip file that includes all the required source files:

$ zip -r hello.zip bin lib
  adding: bin/ (stored 0%)
  adding: bin/hello.rb (deflated 11%)
  adding: lib/ (stored 0%)
  adding: lib/greetings.rb (deflated 7%)
$ unzip -l hello.zip 
Archive:  hello.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
        0  02-14-11 21:01   bin/
       98  02-14-11 21:01   bin/hello.rb
        0  02-14-11 20:59   lib/
       46  02-14-11 20:59   lib/greetings.rb
 --------                   -------
      144                   4 files

Now we need a ruby header for the zip file, which we'll put into header.rb:

#!/usr/bin/ruby -rubygems -x

require "zip/ziprequire"
$:.push $0
require "bin/hello.rb"
__END__

It runs the ruby interpreter, which then loads zip/ziprequire, adds itself (which will be the zip file) to the path, and then requires our main file, bin/hello.rb. We now prepend this header to the zip file using cat header.rb hello.zip > hello_tmp.zip. If you now try to run unzip -l on this file, you'll get:

$ unzip -l hello_tmp.zip 
Archive:  hello_tmp.zip
warning [hello_tmp.zip]:  97 extra bytes at beginning or within zipfile
  (attempting to process anyway)
  Length     Date   Time    Name
 --------    ----   ----    ----
        0  02-14-11 21:01   bin/
       98  02-14-11 21:01   bin/hello.rb
        0  02-14-11 20:59   lib/
       46  02-14-11 20:59   lib/greetings.rb
 --------                   -------
      144                   4 files

We need to fix this zip file to make it valid. Thankfully zip has an option for this:


$ zip --fix hello_tmp.zip --out hellox.zip
Fix archive (-F) - assume mostly intact archive
Zip entry offsets appear off by 97 bytes - correcting...
 copying: bin/
 copying: bin/hello.rb
 copying: lib/
 copying: lib/greetings.rb

We now have a valid zip archive in hellox.zip, we just need to make it executable by running chmod +x hellox.zip. Now your whole application is a single executable file that you can run:


$ ./hellox.zip 
Enter your name:
Mike

Hello, Mike!

Tuesday, February 8, 2011

Distributing a Ruby application as an archive

Let's say you have a Ruby application you've written, and it consists of multiple files that you require inside your code. You want to run this application on some remote machines. To make it easier to deploy this application, you want to distribute it as a single file (archive.) This is possible with the rubyzip gem. Let's say your main application file (myapp.rb) looks like this:

require "lib/mylib"
require "lib/otherlib"
require "vendorlib/something"

# Do stuff.

Normally you might run your application with ruby ./myapp.rb. To run it on another machine, we can use the zip/ziprequire library in rubyzip. First, make a zip file containing all your application files:

zip -r myapp.zip myapp.rb lib vendorlib

Copy myapp.zip to the remote machine, and you can run it like this:

ruby -rubygems -Imyapp.zip -e 'require "zip/ziprequire"' -e 'require "myapp"'

See rubyzip documentation, specifically ziprequire.rb for more information.

Tuesday, January 11, 2011

Using FileMerge with Mercurial

Mac OS comes with a GUI merge tool called FileMerge. This can be used for merges in Mercurial - Mercurial will do this if its internal merge fails. The binary for FileMerge (opendiff) cannot be used as is, so we need to create a small shell script. You can put this script anywhere in your path, and call it hopendiff:

#!/bin/sh
`/usr/bin/opendiff "$@"`

Then modify your ~/.hgrc and add:

[extensions]
hgext.extdiff =

[extdiff]
cmd.opendiff = hopendiff

[merge-tools]
filemerge.executable = hopendiff
filemerge.args = $local $other -ancestor $base -merge $output 

If you already have some of those sections (like [extensions]), then just add the corresponding lines to those sections.

That's it. When you do a merge in Mercurial, it'll open FileMerge if the internal merge fails. You can also use FileMerge for normal diffs by using hg opendiff instead of hg diff.

Thursday, January 6, 2011

Binary and character set safe MySQL dump and restore

To dump some MySQL databases in a binary-safe way:

mysqldump -uroot -p -hdbhost \ 
  --skip-extended-insert \ 
  --default-character-set=binary \ 
  --databases dbone dbtwo dbthree \ 
  --add-drop-database --master-data=2 | gzip > /tmp/dump.sql.gz

Replace --master-data=2 with --lock-all-tables if dumping from a slave, and record the output of SHOW SLAVE STATUS while everything is locked.

To import this dump:

zcat /tmp/dump.sql.gz | mysql -uroot -p -hotherdbhost \ 
   --unbuffered --batch --default-character-set=binary

To set up replication:

Use the Relay_Master_Log_File:Exec_Master_Log_Pos values from SHOW SLAVE STATUS that you recorded. See this page for explanation:

http://www.mysqlperformanceblog.com/2008/07/07/how-show-slave-status-relates-to-change-master-to/

For example:

CHANGE MASTER TO MASTER_HOST = 'abc123.dklfjalddkj.com',
  MASTER_USER = 'rep', MASTER_PASSWORD = 'CHANGEME',
  MASTER_LOG_FILE = 'abc123-bin.003292',
  MASTER_LOG_POS = 283072761;

Thursday, September 2, 2010

Microsoft Dynamics CRM and PHP

I needed to use PHP for two things:

  • Given a lead's email address, retrieve the name, title, phone and email address of this lead's owner.
  • Insert a new lead into the system.

I spent a lot of time searching the web, and saw many posts of people being unsuccessful with using SoapClient, so I didn't even bother trying. The first order of business was to figure out how to login using IFD (Internet Facing Deployment.) I found this page, which explained how to authenticate. It didn't work, but eventually we figured out that our IFD was not configured properly. In the IFD configuration tool, you can't just leave the "local" subnets blank. At least one IP range must be added, even if you want everything to use IFD. We added 127.0.0.1-127.0.0.255 as the "local" range.

I wrote two classes to handle the CRM. These are in the mscrm.php file.

<?
// vim: filetype=php

class MSCRMServiceException extends Exception {}
class MSCRMLoginFailed extends MSCRMServiceException {}

class MSCRMService {
  protected $host, $org, $domain, $user, $password, $curl_handle, $ticket;

  function __construct($crm_host, $organization, $domain, $user, $password) {
    $this->host = $crm_host;
    $this->org = $organization;
    $this->domain = $domain;
    $this->user = $user;
    $this->password = $password;
    $this->init_curl();
    $this->login();
  }

  function __destruct() {
    curl_close($this->curl_handle);
  }

  public function domain_user() {
    return $this->domain . '\\' . $this->user;
  }

  protected function init_curl() {
    $this->curl_handle = curl_init();
    curl_setopt($this->curl_handle, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($this->curl_handle, CURLOPT_CONNECTTIMEOUT, 10);
    curl_setopt($this->curl_handle, CURLOPT_TIMEOUT, 30);
    curl_setopt($this->curl_handle, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
  }

  protected function login() {
    $request = '<?xml version="1.0" encoding="utf-8"?' . '>
      <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <soap:Body>
          <Execute xmlns="http://schemas.microsoft.com/crm/2007/CrmDiscoveryService">
            <Request xsi:type="RetrieveCrmTicketRequest">
              <OrganizationName>' . $this->org . '</OrganizationName>
              <UserId>' . $this->domain_user() . '</UserId>
              <Password>' . $this->password . '</Password>
            </Request>
          </Execute>
        </soap:Body>
      </soap:Envelope>';
    $headers = array(
      'Host: ' . $this->host,
      'Connection: Keep-Alive',
      'SOAPAction: "http://schemas.microsoft.com/crm/2007/CrmDiscoveryService/Execute"',
      'Content-type: text/xml; charset="utf-8"',
      'Content-length: ' . strlen($request)
    );
    curl_setopt($this->curl_handle, CURLOPT_URL, 'http://' . $this->host . '/MSCRMServices/2007/SPLA/CrmDiscoveryService.asmx');
    curl_setopt($this->curl_handle, CURLOPT_HTTPHEADER, $headers);
    curl_setopt($this->curl_handle, CURLOPT_POST, true);
    curl_setopt($this->curl_handle, CURLOPT_POSTFIELDS, $request);

    $response = curl_exec($this->curl_handle);
    $matches = array();
    if(preg_match('/<CrmTicket>([^<]*)<\/CrmTicket>/', $response, $matches)) {
      $this->ticket = $matches[1];
    } else {
      throw new MSCRMLoginFailed;
    }
  }

  public function xml_request($xml, $service, $action) {
    $request = '<?xml version="1.0" encoding="utf-8"?' . '>
      <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <soap:Header>
          <CrmAuthenticationToken xmlns="http://schemas.microsoft.com/crm/2007/WebServices">
            <AuthenticationType xmlns="http://schemas.microsoft.com/crm/2007/CoreTypes">2</AuthenticationType>
            <OrganizationName xmlns="http://schemas.microsoft.com/crm/2007/CoreTypes">' . $this->org . '</OrganizationName>
          </CrmAuthenticationToken>
        </soap:Header>
        <soap:Body>
          <' . $action . ' xmlns="http://schemas.microsoft.com/crm/2007/WebServices">
            ' . $xml . '
          </' . $action . '>
        </soap:Body>
      </soap:Envelope>';
    $headers = array(
      'Host: ' . $this->host,
      'Connection: Keep-Alive',
      'SOAPAction: "http://schemas.microsoft.com/crm/2007/WebServices/' . $action . '"',
      'Content-type: text/xml; charset="utf-8"',
      'Content-length: ' . strlen($request)
    );
    curl_setopt($this->curl_handle, CURLOPT_URL, 'http://' . $this->host . '/MSCrmServices/2007/' . $service . '.asmx ');
    curl_setopt($this->curl_handle, CURLOPT_HTTPHEADER, $headers);
    curl_setopt($this->curl_handle, CURLOPT_POST, true);
    curl_setopt($this->curl_handle, CURLOPT_POSTFIELDS, $request);
    // Ticket goes into a cookie.
    curl_setopt($this->curl_handle, CURLOPT_COOKIE, 'MSCRMSession=ticket=' . $this->ticket);
    $response = curl_exec($this->curl_handle);
    return $response;
  }

  // Retrieves given $attributes for given $entity where $condition_attribute equals $condition_string.
  public function retrieve_multiple($entity, $attributes, $condition_attribute, $condition_string) {
    $attributes_xml = "";
    foreach($attributes as $attribute) {
      $attributes_xml .= "<q1:Attribute>" . htmlspecialchars($attribute) . "</q1:Attribute>\n";
    }
    $xml = '
      <query xmlns:q1="http://schemas.microsoft.com/crm/2006/Query" xsi:type="q1:QueryExpression">
        <q1:EntityName>' . htmlspecialchars($entity) . '</q1:EntityName>
        <q1:ColumnSet xsi:type="q1:ColumnSet">
          <q1:Attributes>
            ' . $attributes_xml . '
          </q1:Attributes>
        </q1:ColumnSet>
        <q1:Distinct>false</q1:Distinct>
        <q1:Criteria>
          <q1:FilterOperator>And</q1:FilterOperator>
          <q1:Conditions>
            <q1:Condition>
              <q1:AttributeName>' . htmlspecialchars($condition_attribute) . '</q1:AttributeName>
              <q1:Operator>Equal</q1:Operator>
              <q1:Values>
                <q1:Value xsi:type="xsd:string">' . htmlspecialchars($condition_string) . '</q1:Value>
              </q1:Values>
            </q1:Condition>
          </q1:Conditions>
        </q1:Criteria>
      </query>
    ';
    $response_xml = $this->xml_request($xml, "CrmService", "RetrieveMultiple");
    $response = array();
    foreach($attributes as $attribute) {
      $matches = array();
      if(preg_match('/<q1:' . $attribute . '[^>]*>([^<]*)<\/q1:' . $attribute . '>/', $response_xml, $matches)) {
        $response[$attribute] = $matches[1];
      }
    }
    return $response;
  }

  // Create an entity.  $attributes should have attributes as keys and their values as values.
  public function create_entity($entity, $attributes) {
    $xml = "<entity xsi:type=\"$entity\">\n";
    foreach($attributes as $k => $v) {
      $xml .= "<$k>" . htmlspecialchars($v) . "</$k>\n";
    }
    $xml .= "</entity>\n";
    return $this->xml_request($xml, "CrmService", "Create");
  }

  // Usage: $options = retrieve_attribute("lead", "leadsourcecode")
  public function retrieve_attribute($entity, $attribute) {
    $xml = '
      <Request xsi:type="RetrieveAttributeRequest">
        <EntityLogicalName>' . $entity . '</EntityLogicalName>
        <LogicalName>' . $attribute . '</LogicalName>
      </Request>
    ';
    $xml = $this->xml_request($xml, "MetadataService", "Execute");
    // Get a list of <Option> tags.
    $matches = array();
    if(!preg_match_all('/<Option>(.*?)<\/Option>/', $xml, $matches)) {
      return null;
    }
    $options_xml = $matches[1];
    $options = array();
    foreach($options_xml as $option_xml) {
      $matches = array();
      if(!preg_match('/<Value[^>]*>([^<]*)<\/Value>/', $option_xml, $matches)) {
        continue;
      }
      $code = $matches[1];
      if(!preg_match('/<Label>([^<]*)<\/Label>/', $option_xml, $matches)) {
        continue;
      }
      $name = $matches[1];
      $options[] = array("code" => $code, "name" => $name);
    }
    return $options;
  }

  public function find_code_by_name($entity, $attribute, $name) {
    $codes = $this->retrieve_attribute($entity, $attribute);
    foreach($codes as $code) {
      if($code["name"] == $name) {
        return $code["code"];
      }
    }
    return null;
  }
}

class MSCRMLead {
  protected $service;

  function __construct($service) {
    $this->service = $service;
  }

  public function find_user($guid) {
    $retrieve_fields = array("firstname", "lastname", "title", "internalemailaddress", "address1_telephone1");
    return $this->service->retrieve_multiple("systemuser", $retrieve_fields, "systemuserid", $guid);
  }

  public function find_lead_by_email($email) {
    $retrieve_fields = array("leadsourcecode", "firstname", "lastname",
                             "jobtitle", "companyname", "industrycode", "websiteurl", "address1_telephone1", "emailaddress1",
                             "address1_city", "address1_stateorprovince", "address1_country", "ownerid");
    return $this->service->retrieve_multiple("lead", $retrieve_fields, "emailaddress1", $email);
  }

  public function find_owner_by_email($email) {
    $lead_info = $this->find_lead_by_email($email);
    if(!$lead_info["ownerid"]) {
      return null;
    }
    $owner = $this->find_user($lead_info["ownerid"]);
    return $owner;
  }

  public function find_leadsourcecode($leadsource) {
    return $this->service->find_code_by_name("lead", "leadsourcecode", $leadsource);
  }

  public function find_industrycode($industry) {
    return $this->service->find_code_by_name("lead", "industrycode", $industry);
  }

  public function create($lead_info) {
    return $this->service->create_entity("lead", $lead_info);
  }
}

?>

You can now retrieve a lead's owner information like this:

require_once("mscrm.php");

$mscrm_service = new MSCRMService($host, $organization, $domain, $user, $password);
$mscrm_lead = new MSCRMLead($mscrm_service);
$owner = $mscrm_lead->find_owner_by_email($lead_email);

Creating a new lead is a little more involved, since you need to look up codes for picklists, like leadsourcecode. You can do something like this:

require_once("mscrm.php");

$mscrm_service = new MSCRMService($host, $organization, $domain, $user, $password);
$mscrm_lead = new MSCRMLead($mscrm_service);

$lead_info = array();

$leadsourcecode = $mscrm_lead->find_leadsourcecode($lead_source);
if($leadsourcecode) {
  $lead_info["leadsourcecode"] = $leadsourcecode;
}

$lead_info["firstname"] = $firstname;
$lead_info["lastname"] = $lastname;
$lead_info["jobtitle"] = $jobtitle;
$lead_info["companyname"] = $companyname;

$industrycode = $mscrm_lead->find_industrycode($industry);
if($industrycode) {
  $lead_info["industrycode"] = $industrycode;
}

$lead_info["websiteurl"] = $websiteurl;
$lead_info["address1_telephone1"] = $phone;
$lead_info["emailaddress1"] = $email;
$lead_info["address1_city"] = $city;
$lead_info["address1_stateorprovince"] = $state;
$lead_info["address1_country"] = $country;

$mscrm_lead->create($lead_info);
Hopefully this will save someone days of headaches.

Wednesday, January 6, 2010

Transparent TCP proxy in ruby (jruby)

This is a transparent TCP proxy. I only tested it in jruby, but I see no reason why it wouldn't work in ruby as well. The proxy is multithreaded, it starts a new thread to handle every connection, up to a limit. The client and the server can talk at the same time, and IO.select is used to figure out who has data to send.
require "socket"

remote_host = "www.google.com"
remote_port = 80
listen_port = 5000
max_threads = 5
threads = []

puts "starting server"
server = TCPServer.new(nil, listen_port)
while true
  # Start a new thread for every client connection.
  puts "waiting for connections"
  threads << Thread.new(server.accept) do |client_socket|
    begin
      puts "#{Thread.current}: got a client connection"
      begin
        server_socket = TCPSocket.new(remote_host, remote_port)
      rescue Errno::ECONNREFUSED
        client_socket.close
        raise
      end
      puts "#{Thread.current}: connected to server at #{remote_host}:#{remote_port}"

      while true
        # Wait for data to be available on either socket.
        (ready_sockets, dummy, dummy) = IO.select([client_socket, server_socket])
        begin
          ready_sockets.each do |socket|
            data = socket.readpartial(4096)
            if socket == client_socket
              # Read from client, write to server.
              puts "#{Thread.current}: client->server #{data.inspect}"
              server_socket.write data
              server_socket.flush
            else
              # Read from server, write to client.
              puts "#{Thread.current}: server->client #{data.inspect}"
              client_socket.write data
              client_socket.flush
            end
          end
        rescue EOFError
          break
        end
      end
    rescue StandardError => e
      puts "Thread #{Thread.current} got exception #{e.inspect}"
    end
    puts "#{Thread.current}: closing the connections"
    client_socket.close rescue StandardError
    server_socket.close rescue StandardError
  end

  # Clean up the dead threads, and wait until we have available threads.
  puts "#{threads.size} threads running"
  threads = threads.select { |t| t.alive? ? true : (t.join; false) }
  while threads.size >= max_threads
    sleep 1
    threads = threads.select { |t| t.alive? ? true : (t.join; false) }
  end
end