Taking ActiveRecord to the Next Level Blythe Dunham

Goal Leverage advanced MySQL functionality with ActiveRecord

Disclaimer!!!! Premature Optimization

ActiveRecord 101 What's going on under the covers?

ActiveRecord 101 class User < ActiveRecord::Base end CREATE TABLE `users` ( `id` int(11) NOT NULL auto_increment, `name` varchar(255) default NULL, `password` varchar(255) default NULL, ` ` varchar(255) default NULL, `created_at` datetime default NULL, `updated_at` datetime default NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 Database Table ff Active Record Model

ActiveRecord 101 with Animals! class Animal < ActiveRecord::Base end CREATE TABLE `animals` ( `id` int(11) NOT NULL auto_increment, `name` varchar(255) default NULL, `password` varchar(255) default NULL, ` ` varchar(255) default NULL, `species_id` int(11) default NULL, `created_at` datetime default NULL, `updated_at` datetime default NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 Database Table ff Active Record Model

Creating a Record animal = Animal.new animal.name = 'Jerry Giraffe' animal.password = 'jerry' animal.save! INSERT INTO `animals` (`name`, `updated_at`, `species_id`, `password`, ` `, `created_at`) VALUES('Jerry Giraffe', ' :48:28', NULL, 'jerry', NULL, ' :48:28')

Updating a Record animal.name = 'Jerry G' animal.save! UPDATE `animals` SET `updated_at` = ' :01:06', `name` = 'Jerry G' WHERE `id` = 1

Finding a Record jerry = Animal.find :first, :conditions => ['name = ?', 'Jerry G'] SELECT * FROM `animals` WHERE (name = 'Jerry G') LIMIT 1 #shortcut Animal.find_by_name 'Jerry G'

Representing Relationships Species name Animal name password fav_beer updated_at created_at species_id

CREATE TABLE `animals` ( `id` int(11) NOT NULL auto_increment, `name` varchar(35) NOT NULL, ` ` varchar(40) default NULL, `fav_beer` enum('Guiness','Pabst','Redhook','Chimay') default 'Pabst', `created_at` datetime default NULL, `updated_at` datetime default NULL, `password` varchar(25) character set latin1 collate latin1_bin NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 Representing Relationships(DDL) CREATE TABLE `species` ( `id` int(11) NOT NULL auto_increment, `name` varchar(255), PRIMARY KEY (`id`), ) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8

Representing Relationships (AR) class Animal < ActiveRecord::Base belongs_to :species end class Species < ActiveRecord::Base has_many :animals end

Representing Relationships (AR) jerry.species SELECT * FROM `species` WHERE (`species`.`id` = 1) species.animals SELECT * FROM `animals` WHERE (`animals`.species_id = 1)

Representing Relationships (AR) giraffe = Species.find_by_name 'giraffe' giraffe.animals << jerry SELECT * FROM `species` WHERE (`species`.`name` = 'giraffe' ) LIMIT 1 UPDATE `animals` SET `species_id` = 1, `updated_at` = ' :15:54' WHERE `id` = 7

Migration Set limits Set default values Identify NOT NULL columns Use enumerated columns Custom DDL Add (unique) indices Foreign Keys are great Primary Key Modifications

Migration 101 ruby script/generate scaffold Animal name:string password:string string fav_beer:string class CreateAnimals < ActiveRecord::Migration def self.up create_table :animals do |t| t.string :name t.string :password t.string : t.string :fav_beer t.timestamps end def self.down drop_table :animals end

Set Limits Default String is VARCHAR(255) create_table :animals do |t| t.string :name, :limit => 35 t.string :password, :limit => 25 t.string : , :limit => 40 t.string :fav_beer, :limit => 40 t.timestamps end

Numeric Type Limits t.integer :mysmallint, :limit => 2 "Smart types" determines numeric type for MySQL :limit Numeric TypeColumn Size 1tinyint1 byte 2smallint2 bytes 3mediumint3 bytes 4, nil, 11int(11)4 bytes 5 to 8bigint8 bytes

Set columns to NOT NULL create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string : , :limit => 40 t.string :fav_beer, :limit => 40 t.timestamps end

Set default values create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string : , :limit => 40, :default => nil t.string :fav_beer, :limit => 40 :default => 'Pabst' t.timestamps end

Remove unneeded columns create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string : , :limit => 40, :default => nil t.string :fav_beer, :limit => 40 :default => 'Pabst' t.timestamps end

Enumerated Column Plugin create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string : , :limit => 40, :default => nil t.enum :fav_beer, :default => 'Pabst' :limit => %w(Chimay Pabst Redhook) t.timestamps end

Think about the table parameters create_table :animals, :options => 'ENGINE=MyISAM' do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string : , :limit => 40, :default => nil t.enum :fav_beer, :default => nil :limit => %w(Chimay Pabst Redhook) t.timestamps end

Custom DDL create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string : , :limit => 40, :default => nil t.enum :fav_beer, :default => nil :limit => %w(Chimay Pabst Redhook) t.timestamps end #case sensitive password (encrypted) execute "ALTER TABLE `animals` ADD `password` varchar(25) character set latin1 collate latin1_bin NOT NULL"

Create (Unique) Indices create_table :species do |t| t.string :name, :null => false, :limit => 25 end add_index :species, :name, :unique => true, :name => 'uk_species_name'

ActiveRecord Uniqueness class Species < ActiveRecord::Base validates_uniqueness_of :name end Doesn't Guaranty Data Integrity!

I Heart Foreign Keys Referential Integrity

The AR Way: Foreign Keys class Species < ActiveRecord::Base has_many :animals, :dependent => :nullify end

The Rails Way: Foreign Keys class Species < ActiveRecord::Base has_many :animals, :dependent => :nullify end Dependent Value SQL Equivalent: :nullify => ON DELETE SET NULL :delete_all => ON DELETE CASCADE :destroy => No SQL equivalent. Every association is instantiated and and callbacks are executed before destruction

Redhills Foreign Key Migration Plugin to the rescue! add_column :animals, :species_id, :integer, :references => :species, :name => 'fk_animal_species', :on_delete => :set_null, :on_update => :cascade ALTER TABLE `animals` ADD `species_id` int(11); ALTER TABLE animals ADD CONSTRAINT fk_animal_species FOREIGN KEY (species_id) REFERENCES species (id) ON UPDATE CASCADE ON DELETE SET NULL;

CREATE TABLE `animals` ( `id` int(11) NOT NULL auto_increment, `name` varchar(35) NOT NULL, ` ` varchar(40) default NULL, `fav_beer` enum('Guiness','Pabst','Redhook','Chimay') default 'Pabst', `species_id` int(11) default NULL, `created_at` datetime default NULL, `updated_at` datetime default NULL, `password` varchar(25) character set latin1 collate latin1_bin NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 Primary Keys CREATE TABLE `species` ( `id` int(11) NOT NULL auto_increment, `name` varchar(255), PRIMARY KEY (`id`), ) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8

Modify the Rails Primary Key Change type with change_column MySQL Migration Optimization Plugin create_table :animals, :primary_key => "special_key", :primary_column => { :type => :integer, :limit => 2, :precision => :unsigned, :scale => 3 } do |t| end CREATE TABLE `animals` (`special_key` smallint(3) UNSIGNED NOT NULL auto_increment PRIMARY KEY) ENGINE=InnoDB;

Advanced ActiveRecord Insert and update options Import bulk data Finder Options Tools: plugin AR-Extensions

ActiveRecord on Steroids: ar- extensions plugin Additional Create and Update options save(options={}) save!(options={}) create(args, options={}) create!(args, options={}) Options :ignore :on_duplicate_key_update :keywords :reload :pre_sql :post_sql

:ignore Standard ActiveRecord: Create the record if it does not already exist unless Animal.find_by_name('Jerry G') Animal.create!(:name => 'Jerry G', :password => 'jerry') end

:ignore Ignore duplicates! One query, less code, fewer queries! Animal.create!({:name => 'Jerry G', :password => 'jerry'}, {:ignore => true})

:on_duplicate_key_update Update the record if it exists, if not create a new one. A lot of code to upsert and performs two SQL queries! jerry = Animal.find_by_name 'Jerry G' jerry ||= Animal.new(:name => 'Jerry G') jerry.password = 'frenchfry' jerry.save!

:on_duplicate_key_update jerry = Animal.new :name => 'Jerry G', :password => 'frenchfry' jerry.save! :on_duplicate_key_update => [:password, :updated_at] INSERT INTO animals (`name`, `updated_at`, `species_id`, `password`,` `, `created_at`) VALUES('Jerry G', ' :17:51', NULL, 'frenchfry', NULL, ' :17:51') ON DUPLICATE KEY UPDATE `animals`.`password`=VALUES(`password`), `animals`.`updated_at`=VALUES(`updated_at`)

Reloading the instance AR Data can become inconsistent with DB after an IGNORE, UPDATE, or ON DUPLICATE KEY UPDATE reload executes more queries For UPDATE the duplicate is automatically reloaded jerry. = jerry.save! :on_duplicate_key_update => [:password, :updated_at], :reload => true, :duplicate_columns => [:name]

More Customization jerry.save(:keywords => 'LOW_PRIORITY', :pre_sql => '/*Pre comment*/', :post_sql => "/*#{__FILE__} #{__LINE__}*/") /*Pre comment*/ UPDATE LOW_PRIORITY `animals` SET `created_at` = ' :13:48', `species_id` = NULL, ` ` = NULL, `password` = 'frenchfry', `updated_at` = ' :45:38', `name` = 'Jerry G' WHERE `id` = 7/*animal_controller.rb 147 */

Import (Bulk Insert) Instead of one-by-one, insert a ton of records fast

Import (Bulk Insert) Standard way: Insert each animal one by one Animal.create!(:name => 'dolly dolphin', :password => 'dolly') Animal.create!(:name => 'big birdie', :password => 'birdie') and so on….

Fast Import: One INSERT animals = [ Animal.new(:name => 'dolly dolphin', :password => 'dolly'), Animal.new(:name => 'big birdie', :password => 'birdie')] Animal.import animals INSERT INTO `animals` (`id`,`name`,` `,`fav_beer`,`created_at`,`updated_at`,`password`) VALUES (NULL,'dolly dolphin',NULL,'Pabst', ' :17:15',' :17:15','dolly'), (NULL,'big birdie',NULL,'Pabst', ' :17:15',' :17:15','birdie') ON DUPLICATE KEY UPDATE `animals`.`updated_at`=VALUES(`updated_at`)

Fastest Import: fewer columns columns = [ :name, :password ] values = [['dolly dolphin', 'dolly'], ['big birdie', 'birdie']] options = {:validate => false, :timestamps => false} Animal.import columns, values, options INSERT INTO `animals` (`name`,`password`) VALUES ('dolly dolphin','dolly'),('big birdie','birdie')

Insert Select Standard: Query and Insert one by one Species.find(:all).each do |s| SpeciesZoo.create!(:species_id => s.id, :zoo_id => zoo.id, :extra_info => 'awesome') end Executes a query for each species INSERT INTO `species_zoos` (`zoo_id`, `id`, `species_id`, `extra_info`) VALUES (1, 3, 3, 'awesome') INSERT INTO `species_zoos` (`zoo_id`, `id`, `species_id`, `extra_info`) VALUES (1, 3, 2, 'awesome') And so on…

Insert Select Import SpeciesZoo.insert_select( :from => :species, :select => ['species.id, ?', zoo], :into => [:species_id, :zoo_id]) One INSERT statement INSERT INTO `species_zoos` ( `species_id`, `zoo_id` ) SELECT species.id, 1 FROM `species`

Temporary Tables Not so good for slave replication Can be used as a sandbox then imported into a real table with ar-extensions gem Animal.create_temporary_table do |t| t.create!(:name => 'giraffe', :password => 'goo') Animal.insert_select( :from => t, :select => [:name, :password, :fav_beer], :into => [:name, :password, :fav_beer], :on_duplicate_key_update => [:password, :fav_beer]) end

Customizing Find Additional finder options :keywords :pre_sql :post_sql :index_hint

Customizing Find Animal.find(:all, :conditions => ['name = ?', 'Jerry G'], :keywords => 'HIGH_PRIORITY', :pre_sql => '/*Pre comment*/', :post_sql => 'FOR UPDATE /*After the fact*/', :index_hint => 'USE INDEX (uk_animal_name)' ) /*Pre comment*/ SELECT HIGH_PRIORITY * FROM `animals` USE INDEX (uk_animal_name) WHERE (name = 'Jerry G') FOR UPDATE /*After the fact*/

Need more? Get dirty with find_by_sql sql = Animal.send :finder_sql_to_string, :conditions => ['name = ?', 'Jerry G'] sql.gsub! /WHERE/, 'where /* Dirty hand */' Animal.find_by_sql sql

More: find_union & count_union Animal.find_union( {:conditions => ['animals.name like ?', 'Jerry%']}, {:conditions => ['species.name = ?', 'giraffe'], :include => :species} ) (SELECT `animals`.* FROM `animals` WHERE (animals.name = 'Jerry G')) UNION (SELECT `animals`.* FROM `animals` LEFT OUTER JOIN `species` ON `species`.id = `animals`.species_id WHERE (species.name = 'giraffe'))

Finder Issues: Speed and Memory paginate - less loaded into memory :select option - Data is retrieved faster when fewer columns are selected

Paginated Finders Rails includes :batch_size option Animal.find_each(:batch_size => 2) do |animal| #do something end Will Paginate Plugin page = 1 begin animals = Animal.paginate :per_page => 2, :page => page animals.each{|animal| …do something… } end while (page = animals.next_page) Paginating Find Plugin

:select Data is retrieved faster when fewer columns are selected Animal.find :first, :select => 'name'

:include hates :select But :select is ignored with eager loading (:include)! Animal.find :first, :select => 'animals.name, species.name', :include => :species, :conditions => ['species.name like ?', 'giraffe'] SELECT `animals`.`id` AS t0_r0, `animals`.`name` AS t0_r1, `animals`.` ` AS t0_r2, `animals`.`fav_beer` AS t0_r3, `animals`.`created_at` AS t0_r4, `animals`.`updated_at` AS t0_r5, `animals`.`password` AS t0_r6, `animals`.`species_id` AS t0_r7, `species`.`id` AS t1_r0, `species`.`name` AS t1_r1 FROM `animals` LEFT OUTER JOIN `species` ON `species`.id = `animals`.species_id WHERE (species.name like 'giraffe') LIMIT 1

Alternatives to Eager Loading Eager loading for sparse :include can be time consuming Use :join instead of :include Eager Load Plugin Rails Query Cache helps ActiveRecordContext Plugin PiggyBack Plugin

:join instead of :include Eager loading is slow in Rails and can be slow on the database. Use an (inner) :join instead of :include animal = Animal.find :first, :select => 'animals.name, species.name as spec_name', :joins => :species, :conditions => ['species.name like ?', 'giraffe'] animal.spec_name == 'giraffe'

Force it with Eager loading plugins Eager loading is slow in Rails and can be slow on the database. eload-selecteload-select plugin

Help from Rails 2 Query Cache Animals of the same species are only loaded once ActiveRecord::Base.cache { Animal.find(:all).each {|a| a.species } } Animal Load (1.8ms) SELECT * FROM `animals` Species Load (0.3ms) SELECT * FROM `species` WHERE (`species`.`id` = 2) CACHE (0.0ms) SELECT * FROM `species` WHERE (`species`.`id` = 2) Species Load (0.3ms) SELECT * FROM `species` WHERE (`species`.`id` = 1) CACHE (0.0ms) SELECT * FROM `species` WHERE (`species`.`id` = 1)

ActiveRecordContext Plugin ActiveRecord::Base.with_context do animals = Animal.find :all Species.prefetch animals animals.each {|a| a.species } end Animal Load (0.4ms) SELECT * FROM `animals` [Context] Storing Animal records: 1, 2, 3, 4, 5, 6, and 7 Species Load (0.4ms) SELECT * FROM `species` WHERE (`species`.`id` IN( 2,1 )) [Context] Storing Species records: 2 and 1 [Context] Found Species #2 [Context] Found Species #1

Piggyback Plugin Delegate records with :has_one and :belongs_to associations Great for accessing extension tables with TEXT or BLOB

Piggyback Plugin Uses joins to delegate records from :has_one and :belongs_to associations class Animal < ActiveRecord::Base belongs_to :species piggy_back :species_name, :from => :species, :attributes => [:name] end animal = Animal.find :first, :piggy => :species_name animal.species_name == 'giraffe' SELECT animals.*, species.name AS species_name FROM `animals` LEFT JOIN species ON species.id=animals.species_id LIMIT 1

Avoiding Deadlock Deadlock Retry Deadlock Retry plugin - retries query up to 3 times Batch Operations (AR-Extension plugin) Animal.delete_all(['name like ?','giraffe%'], :batch_size => 50)

Reading from a Replica Masochism plugin

Export Data ar-fixtures: Export entire table to yaml ar-extensions: export to csv ar-dumper: export (paginated) to yaml, xml or csv

Cache! Show me the money! Query Cache Memcache Static Record Cache plugin

ActiveRecord plugins and gems AR-Extensions Piggy Back Eager Loading Active Record Context weenie.net/projects/plugins/active_record_context/ weenie.net/projects/plugins/active_record_context/ Will Paginate Deadlock Retry Paginating Find Static Record Cache

Other Useful Plugins Migration Plugins Redhills Consulting Foreign Key Migrations Enum Column MySQL Migration Optimizer Replica Plugins Masochism Active Delegate connections-in-ruby-on-rails connections-in-ruby-on-rails Export Plugins Ar-fixtures Ar-dumper

Questions? Thanks for attending! I hope you are on the next level. Slides available at:

Pics Biz card: Toolbox: Cash: Emoo cow: Giraffe in Car: Leafcutters: Puppy: Dump truck: Ignore sign: Tigers: Single Leafcutter: Giraffe with tongue: Giraffe in a can: Beaver: Mini Pig: Giraffe herd: Giraffe skin: Dirty hands: Blythe twins: Dead lock: Foreign Keys: Gorilla: Speed of light: Fat Giraffe: