DataMapper 0.10+ Basic Sphinx Support
I needed sphinx support and I needed it now. Apparently dm-sphinx-adapter wasn't ported to dm 0.10. I started porting it, but I kind of felt like I was going down a rabbit whole. I will continue to work on it over the next couple days, in the meantime at Vokle we used this to get our fulltext on. Note, this is doing the fulltext search across all columns in your fulltext index. Its not doing cool matchers like :updated_at.gt yada-yada. It also doesn't implement a DataMapper Adapter, its a super hack, but it gets the job done for us.
Give me your two cents and stay tuned for a full blown adapter.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | module DataMapper module Model def sphinx_setup(opts={}) @_sphinx_index = opts[:index] end def search(_query_, opts={}) sph_client = opts.delete(:client) || Riddle::Client.new(*SimpleSphinx.conf) sph_client.match_mode = opts.delete(:match_mode) || :extended sphinx_query = _query_.to_s.gsub(/[\(\)\|\-!@~"&\/]/){|char| "\\#{char}"} results = sph_client.query(sphinx_query, @_sphinx_index || self.storage_names[:default].to_sym) matches = results[:matches] ids = matches.map{|match| match[:doc]} # Fetch self.all(opts.merge({:id => ids})) end end end class SimpleSphinx class << self def conf [server, port] end def server=(s) @_server = s end def port=(p) @_port = p end def server @_server || 'localhost' end def port @_port || 3312 end end end |
Wanna use it?
1 2 3 4 5 6 7 8 9 10 | # Your Model here Person.sphinx_setup(:index => "indexName") #defaults to storage name SimpleSphinx.server = "example.com" SimpleSphinx.port = 3312 Person.search("cory") #=> DataMapper::Collection # you can pass an additional hash of stuff you would pass to Model#all and this will be used to filter # the result set down POST sphinx when its collecting data from datamapper Person.search("cory", :limit => 10, :gender => :male) |
You've got your sphinxter back. Congrats!
DataMapper and Merb, sharing your errors via the merb display API.
This is a quick little snippet. At Vokle all of our products are built using our core API, which thanks to merb, was a piece of cake. The thing that sucks is sometimes we return objects as JSON via the display API and the "errors" are missing in the event that validation failed. How to fix that?
Throw this somewhere:
1 2 3 4 5 6 7 8 9 10 | module DataMapper module Validate class ValidationErrors def to_json @errors.to_hash.to_json end end end end |
Now in your merb controllers, when you are displaying an object that may have errors:
1 2 3 4 5 6 7 8 | class People < Merb::Controller def create(person) #... *SNIP* ... display @person, nil, {:methods => [:errors]} #... *SNIP* ... end end |
And whatever is getting your data back in XML or JSON (yeah or YAML, right) will get the errors on your object as well. Cool.
Yay, users give you invalid data. Congrats.
DM Cutie, et al. 0.4 release (minor updates)
I released dm-cutie, dm-cutie-ui, and dm-cutie-extras with some minor releases on gemcutter.org today.
The minor updates include a lot of help information and some tweaks to dm-cutie-ui to make it easier to use/understand (I hope).
+ add 'help' area describing all the default views
+ MysqlIndex/ Steps column info in help
+ Sort Tables Repo, Gen, Executed, Others by alpha (routes key should be view_name)
+ menu cleanup
+ @records count on page
+ Moved connected to
+ Changed word on relationship link
+ Format of page title
+ Removed duplicate column on mysql_index
+ Sort buttons moved
If you haven't installed DM Cutie, make sure your rubygems is up to date and do:
1 2 3 4 5 | gem install gemcutter --source http://gemcutter.org gem tumble gem install dm-cutie -v 0.4.0 gem install dm-cutie-ui -v 0.4.0 gem install dm-cutie-extras -v 0.4.0 |
Information on setting DM Cutie up is in this post.
If you want the source its available on my github page.
I'll be adding some more interesting tracking stuff for MySQL in the next few days. Now that Vokle has officially launched, I find myself with more time to work on my projects.
When I started this project I really wanted it to be a cool way to get information on all your DM storage needs. I am now realizing that DM Cutie should probably only support SQL and DO backed stores, and I'll be pairing down the internals over the next few days with that in mind.
Most of my database experience as far as optimizing goes is in MySQL, if anyone has some PostgreSQL or SQlite awesomeness they'd like to share, I'd really appreciate it.
DataMapper Cutie – The query tracker and profiler
So Ive been busting my butt on this query tracker for a while now. Its been finished twice and not released (once when I hated the API, and once right before 0.10 was release). But here she is, open for the hacking.
Its pretty easy to get set up, the gems are on http://gemcutter.org
To get started:
1 2 3 4 5 6 7 8 | $ gem sources -a http://gemcutter.org # Add gem cutter to your gem sources $ gem install dm-cutie # This is a plugin pack for dm-cutie $ gem install dm-cutie-extras # this is the front-end to dm-cutie $ gem install dm-cutie-ui |
If you plan to use MySQL to store dm-cuties information, set up a schema for that now by doing:
1 | mysql: CREATE SCHEMA `dm_cutie`; |
dm-cutie will allow you to store stuff across different adapters, so you can actually profile a Mysql DataMapper Repository into a dm-cutie sqlite3 repository. Its ok, she gets it
Now, in your application after your default DataMapper repository is set up do:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | require 'dm-cutie' #currently only data_objects is supported DataMapper::Cutie.enable_adapter :data_objects require 'dm-cutie-extras' # Query Plans! DataMapper::Cutie::Extras.load :mysql_execution_step # Index optimization tips! DataMapper::Cutie::Extras.load :mysql_index # Mysql Errors and warnings! DataMapper::Cutie::Extras.load :mysql_warning DataMapper::Cutie.setup do |c| c[:repo_name] = :dm_cutie # name of your repo c[:exclude_models] = false #[:person, :car] #models to exclude c[:only_models] = false #[:article, :address] #model to include c[:slow_query_length] = 2 end #Forces Cutie to always migrate HER models DataMapper::Cutie.start(true) |
Let dm-cutie sit in your code for a while, she will start to generate lost of data on your repository. I would recommend not putting dm-cutie in a production environment due to the amount of queries that it generates. It is best used in a development or a staging environment and then applies the learned optimization strategies to your production environment.
After cutie has been running a while (or immediately if you are impatient) from the command line do:
1 | $ dm-cutie-ui -p 4567 --extras=mysql_warning,mysql_index,mysql_execution_step |
If you didn't use any of the extras in your application you SHOULDNT list them when you start the dm-cutie-ui.
That's that. You just need to sign in using the URI for your repository, it will be something like: mysql://root@localhost/dm_cutie.
An important note is that when you log into dm-cutie-ui you are logging into the dm-cutie repository, not your applications, so keep that in mind so you dont get errors saying the dm-cutie repo doesn't exist.
Well that's it for now, I'll post about this more after the holiday when I make a few more plugins. Enjoy!
There is a ton more documentation at:
DataMapper is Viewable
Even when I'm panicking I'm a fan of datamapper. I like how clean it generally makes my code, but I do notice that I tend to have the same queries cropping up from time to time either in the same application or in other apps using my model library. This just wasn't dry enough for me, so I made dm-is-viewable. It gives sql-like view functionality to DataMapper Resources. It's a pretty simple plugin, but lets take a look, um kay?
Let's start with a really simple User class...
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | class User include DataMapper::Resource belongs_to :location property :id, Serial property :name, String property :username, String property :password, String property :gender, String property :age, Integer property :favorite_color, String property :favorite_number, Integer end class Location include DataMapper::Resource has n, :users property :id, Serial property :name, String property :desc, String end |
Making your resource viewable is pretty easy, first grab the dm-is-viewable gem, then say that your resource is viewable. Views take the exact same parameters that you could pass to Resource.all. The only difference is that dm-is-viewable stores them until they are called.
1 2 3 4 5 6 7 8 9 10 | class User include DataMapper::Resource is :viewable #lets create some views create_view :legal_women, :gender => 'female', :age.gt => 18 create_view :serial_killers, :favorite_number => 666, :favorite_color => 'black' #... Resource code *SNIP SNIP* end |
See the befores and afters below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | # Legal Women Before User.all(:gender => 'female', :age.gt => 18) # Legal Women After User.view :legal_women # Serial killers before User.all(:favorite_number => 666, :favorite_color => 'black') # Serial killers after User.view :serial_killers # You can also pass further limiting query parameters to #view. # Legal Local Women Before User.all(:gender => 'female', :age.gt => 18, User.location.name => 'Los Angeles') # Legel Local Women After User.view :legal_women, User.location.name => 'Los Angeles' # Passing additional parameters FURTHERS the limitation of records, so.. User.view :legal_women, :gender => 'male' # => Would return nil, the query would essentially be generated as: # SELECT * from users where gender = 'male' and gender = 'female' |
Simply, clean, useful. Woot.
Massive Flaw in DataMapper 0.9.6 – Write Once Read Many Many Many Many Many, Im outta space here…
So I found a pretty massive flaw in DataMapper (ticket @ lighthouse). Whenever using the :fields attribute when doing Resource#first or Resource#all a crap load (techincally speaking) of queries are fired at MySQL. I put a ticket in and a lot of information and links to some pasties with examples (see lighthouse). I don't have the time to look into this now, being that I have a deadline Saturday, but it's a pretty serious problem and I'm sure a lot of people are experiencing it and don't realize it.
To try it out for yourself lets use Merb (and Ive tried it without merb, and without any dm plugins and the problem still persists).
Steps to reproduce:
1 2 3 | merb-gen app my_test_app cd ./my_test_app merb -i |
1 2 3 4 5 6 7 8 9 | DataMapper.auto_migrate! #turn on dm loggering (thats what I call it) DataObjects::Mysql.logger = DataObjects::Logger.new('log/dm.log', 0) # create a few users, then, get their IDs, I dare you... User.all(:fields=>[:id]) |
So, the problem is if you look at your mysql log or DataMapper Query Log you'll see a crapload of selects trying to get what the first select in the list retrieved. The funnier part is, the subsequent selects are selecting the fields that aren't included in the :fields list.
Why is this a huge problem? That query above for my 3 user test table generated about 90 trips to MySQL. The 'friend button' on Vokle.com just generated about 240 trips to MySQL in one click, that's how I found the issue in the first place.
Someone fix this please!? If its not fixed by Saturday (my day 'off'), I'm going to look into it.
Happy (bug) hunting.
Modular Tables? DataMapper is Remixable.
One of my favorite things about Ruby is mixins. I love the pseudo multiple inheritance it provides. One thing that is frustrating to me though when working on developing back-ends is the amount of un-DRY code I am forced to create when working with the database.
For example, comments. Look at sites like MySpace, Facebook, and SocialGiant3.0 (sarcasm), everything is commentable. A user can comment on pictures, profiles, videos, and just about any other resource you can think of.
In this scenario you generally have one of two choices:
A.) You create a comments table for each of the resources: user_comments, profile_comments, etc.
B.) You create a giant comments table and horizontally partition it based on the comment_type.
I prefer the choice 'A', but the downfall, a very wet, anti-DRY set of models. This is were dm-is-remixable comes in. It gives you the ability to remix datamapper models into other models.
Whats that mean?
You write one model describing what a comment looks like and you mix it into User, Profile, etc; like magic, you've got a dry 'comment' model and a nice normalized table structure to store it in.
Here are some examples:
Create a remixable database model named Commentable
module Commentable
include DataMapper::Resource
is :remixable,
:suffix => "comment"
property :id, Integer, :key => true, :serial => true
property :comment, String
property :created_at, DateTime
end
Create a remixable image model and mix it into User
module Image
include DataMapper::Resource
is :remixable
property :id, Integer, :key => true, :serial => true
property :description, String
property :path, String
end
class User
include DataMapper::Resource
property :id, Integer,
:key => true,
:serial => true
property :first_name, String,
:nullable => false,
:length => 2..50
property :last_name, String,
:nullable => false,
:length => 2..50
remix n, :images, :as => "pics" #:as is optional and gives you an additional accessor
end
user = User.new
user.user_images
=>[] #Array of images
user.pics
=>[] #Array of images
Enhance the functionality of UserImages
class User
enhance :images do
def dimensions
# ... get the dimensions of the image
end
def resize(x,y)
# ... resize the image
end
end
end
user = User.first
user.pics.first.dimensions
=> "300x500"
user.pics.first.resize(100,300)
=> "100x300"
Remix commentables into Videos
class Video include DataMapper::Resource is :remixable property :id, Integer, :key => true, :serial => true property :description, String property :path, String remix n, :commentables, :as => "comments" # as is optional, provides add'l accessor name end video = Video.new video.comments =>[] # Array of comments video.video_comments =>[] # same array of comments
Remixing commentables into User;
class User # ... from declaration above ... # This is some of the more complex syntax # here we are remixing a module between two of the same class, so we need some sort of alias # :as => is the accessor method # :for => is the class that this class remixes the module with # :via => what we want to call the other user object remix n, :commentables, :as => "comments", :for => "User", :via => "commentor" end commentee = User.first commentor = User.get(2) comment = UserComment.new comment.comment = "Zomg! I talk non-stop, you are awesome!" comment.commentor = commentor commentee.comments << comment commentee.save
How do you get it? Well its in dm-more. If you dont know what DataMapper is, then git with the program.
From your favorite command line:
git clone git://github.com/sam/dm-more.git
Then from your source:
require 'dm-is-remixable'